Amalgamation of BDD, parallel execution and mobile automation
Download as PPTX, PDF0 likes3,631 views
The document discusses an integrated strategy for mobile automation testing that combines behavior-driven development (BDD), parallel execution, and effective tool selection for improved software quality. It highlights the challenges in mobile automation such as device fragmentation and communication issues, while proposing mitigation strategies like automated customer-centric testing and continuous integration. The authors, Jigesh Shah and Mradul Kapoor, emphasize the importance of collaboration among stakeholders and the implementation of best practices to enhance testing efficiency and effectiveness.
Amalgamation of BDD, parallel execution and mobile automation
- 1. Amalgamation of Behavior driven development (BDD), parallel execution and mobile automation 1 - Jigesh Shah - Mradul Kapoor
- 2. 2 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. Author’s profile Mradul Kapoor Mradul Kapoor is working in one of the leading consulting firm based in Mumbai, India. His experience, knowledge and skills include Business Analysis, Automation and Functional Testing. Mradul has worked on various testing tools like QTP, Selenium, Oracle Oats, Coded UI, SpecFlow, SpecLog, Appium, Microsoft team system and various test management tools. He has over 7.5 years of experience in US Healthcare, Insurance, eLearning, Life Science and Human and Health Services Jigesh Shah Jigesh Shah is working in one of the leading consulting firm based in Mumbai, India. His experience, knowledge and skills include Business Automation and Functional Testing. Jigesh a Certified Java Programmer has worked on various testing tools like QTP, Selenium, QC, Rally, JIRA , SpecFlow, SpecLog, Appium, Microsoft team system and various test management tools. He has over 7.5 years of experience in US Healthcare, Insurance, and eLearning
- 3. 3 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. Preface With the rise of Mobile first approaches, a substantial traffic has now shifted from the desktops to the Mobile phones. To ensure quality in mobile apps, Testing needs to be continuous and inclusive. This makes Mobile Test Automation a formidable business driver. Along with that there needs to be strong collaboration between developers, testers and non-technical or business participants in a software project to maintain application consistency across multiple platforms. The intent of this presentation will be to provide a comprehensive mobile automation testing strategy using Behavior Driven Development, optimized selection of target devices, and an effective combination of manual and automated testing; use of correct tools to cover appropriate functional testing essential for getting your mobile applications to market on time and within budget. This presentation will also discuss on how the scenarios which are derived from the requirement can be automated using BDD approach across Web and Mobile. Also, it will highlight on how parallel execution can be achieved using BDD & Mobile Automation. Target audience : Senior Leadership, Head Testing Practice, Head Quality Assurance, Delivery Head, Test Managers, Test Leads and Test Professionals
- 4. 4 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. Current market analysis and trends With the fantastic growth of mobile computing platforms there has been a dramatic increase in the value of mobile applications for most companies. One of the biggest challenges that one faces when developing a mobile application is how to test it. Period Samsung Apple Lenovo Huawei Others Q1 2015 24.60% 18.30% 5.60% 5.20% 46.30% Q1 2014 30.70% 15.20% 6.60% 4.70% 42.80% Q1 2013 31.50% 16.90% 4.70% 4.20% 42.70% Q1 2012 28.90% 22.90% 5.00% 3.40% 39.80% Period Android iOS Windows Phone BlackBerry OS Others Q1 2015 78.00% 18.30% 2.70% 0.30% 0.70% Q1 2014 81.20% 15.20% 2.50% 0.50% 0.70% Q1 2013 75.50% 16.90% 3.20% 2.90% 1.50% Q1 2012 59.20% 22.90% 2.00% 6.30% 9.50% Source : IDC May 2015
- 6. 6 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. What are the testing challenges faced by today’s organizations while automating mobile and web automation? Mobile automation challenges Outcome Parallel execution Rapid rollout schedule • Rapid rollouts of bug fixes • Constant expectation to meet Application quality Device fragmentation • Issues with multiple Devices and Browsers • Object Identification support issues Mobile lab • Difficult to test in the field • No VPN on old devices • How to push updates and test them? Mobileautomation Behaviordrivendevelopment Communication issues • Issues with understanding the requirements • Each team has developed their own understanding mechanism, making communication ineffective Testing challenges • Identify devices to test on to support your user base • Identify device limitations and characteristics Right tools & approach • Applications supported • Lack of Frameworks • Low Automation expertise Lack of documentation • Documentation is neglected at times • Document not inline with the code changes • Delays in timely decisions • Lack of proper communication channel • No correct ROI of automation • High script maintenance • Low resource moral • No correct channel is followed for coding standards
- 8. 8 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. Common approaches/options Description Benefits Challenges Web automation, manual on mobile • All manual don’t automates • No additional cost for transition or to build Mobile Automation Framework • No need for Mobile Lab Setup • No additional learning required • Does not address underlying issues • Risk of credibility and brand value • Loss of revenue and business Web and mobile automation, no parallel execution • Automate Web and Mobile Application without Continuous Integration and Parallel Execution • Efficient Testing • Upgradation and Reusability • User Environment Simulation • Reduces Human Error • May result into code redundancy for mobile and web scripts • Higher transition cost • mobile technologies changes too fast – automation need continuous support. • May result into inconsistency Recommended approach Description Benefits Challenges Automated customer centric parallel execution • Automate Web and Mobile Application including Customer Centric Solution, Continuous Integration and Parallel Execution • Thinking through solutions, before starting to code • Documentation of Assumptions and Specifications • Addresses Device Fragmentation issue • Remote and cloud automation • Test consistently. Each test is executed in the same way • Active involvement from Business and Technical Team • Resistance to change • Mobile technologies changes too fast – automation need continuous support Here are some key options… Mitigation strategies
- 9. Approach to overcome the issues and challenges
- 10. 10 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. Architecture and design Object repository • Captures the unique identifier to handle the object Configuration files • Stores the environment and other global variables to determine the config values Data repository • Handles the Application under Test data and dynamic data Controller scripts Function Libraries • Reusable page/module level functions Reporting module • Reports and Logs Generation Recovery scripts • Contains the logic to handle exceptions Common utilities • Reusable functions for Native and Web Apps Test scripts • Contains the actual Test Execution Steps Scenarios • Given Set of Preconditions • When Set of Test Condition Mobile and web applications • Then Set of Expected Results BDDWeb and mobile automation
- 11. 11 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. Continuous integration for mobile and web automation Execute Automated Scripts Create mobile project for testing : QA creates the scripts and submits it to the Code Repository Tester Stakeholders Execution on node machines Hub sends the tests to each node and when it is executed the result is sent back to the Hub machine Source code repository CI Servers Hub machine Run deployment scripts on hub CI Server compiles and builds scripts and sends it to the Hub Machine Continuous reporting The Automation Test Execution results are sent to QAs and Stakeholders Send test results The Hub Machine sends the consolidated result back to the CI Server
- 13. 13 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. These elements can serve as a foundation to build an efficient solution Business case study Category Impact Test Efficiency • Ran the Test Suite which comprised of 20 Test Scripts on Mobile and Web Application concurrently. • Out of these 80 Test scripts(20* 4) , 96.25 % scripts passed Test Coverage • Developed automation test suite comprising of 20 test scripts covering 80% of application screens on both Mobile and Web • Also covered 80% of the gestures supported by the Mobile Automation Tool Effort Saving and Flexibility • Test Execution time for Manual Test Cases is around 20 Test Cases * 4 Devices * 15 min = Approx. 1200 min/Cycle =20Hr/cycle • Test Execution time for Automation Test Cases is around 20 Test Cases * 4 Devices * 8 min = Approx. 640 min/Cycle =11Hr/cycle. • Executing parallel on 4 nodes will again reduce the time by 11 / 4 = ~ 3 hours Enhanced Validation • Defects are detected early whenever a new build is deployed • Recovery scenarios helps the Test Cases to proceed ahead in case of any unwanted failures • Details trace of the test is also provided for analysis which can help in concluding an issue. Reusability and Extensibility • Using BDD Components, the same scenario can reused across multiple device with minimal effort. • The Web scripts that were used to automate the Web Application have been extended to incorporate the code for Mobiles
- 15. 15 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. Benefits/value delivered • Reduced coding & maintenance efforts as the tests runs on multiple platforms • Reaping the rewards in the future as we can add more test cases or update existing application flows. Common Code Base • The Mobile Device Lab mandates a particular process for adding devices to the lab. And also makes the lab easily scalable. Common processes can be set for tasks such as resetting installing/uninstalling software on the devices, software or OS upgrade, etc. • It helps in drastically reducing the time required to get the test results Framework and Programming scalability • Everyone in the chain knows exactly what test cases are being executed and the test data being used, as the test cases are written in Gherkin (Given-When-Then) format which are easily readable and can be understood by everyone involved in the SDLC • QAs can just pick these tests and plug them into the framework saving the time & effort of re- writing automation scripts. Automation Execution transparency to Customer • Testing results are more reliable since we test on devices. • Issues can be reported in the software encountered for low bandwidth, security, performance, etc. Testing on Real Devices • Automation runs on real devices can be kicked off immediately without any manual intervention or waiting for the QA team to start the process. • Parallel execution of test suites can be easily configured and fired under CI tool Continuous Testing
- 17. 17 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. There are four phases to implement BDD with mobile automation Implementation approach Feasibility Study and POC • Benefits of automation in this project? • Identifying parts that can be automated on mobile and integrate with web Pilot • Implement the POC along with BDD • Identify challenges and scope of optimization Implementation • Implement the framework along with parallel execution • Integrate mobile and web automation effectively • Add BDD along with required traceability Rollout and Optimize • Rollout the implementation across project • Optimize the common components based on learning from different projects • Rollout the updated implementation
- 18. Best practices and key takeaways
- 19. 19 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. Lessons learned and best practices Shift focus from Tests to Behavior : It helps to describe behaviors in a single notation which is directly accessible to domain experts, testers and developers, so as to improve communication Invest in Continuous Integration : Having an automated job that will build and deploy applications on the mobile, trigger automation runs, email notifications, etc. would really help in achieving a 24/7 service. Invest in Mobile Labs : Testing on real devices helps to uncover some critical bugs and failures and also helps to determine the actual behavior of the devices in various conditions Use Page Object Patterns : It helps the operations and flows in the UI to be separated from verification. This concept makes the code cleaner and easy to understand. Develop Strong Test Recovery : Develop agents that can restart test components on devices. Enhance test case recovery, capability to restart test case.
- 20. 20 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. For organizations facing the challenges related to merging of multiple technologies and platforms, the customer centric solution amalgamated with parallel execution and Mobile Automation will help to address these needs. If implanted properly it can help Organizations to achieve • Collaboration between Business stakeholders, Business Analysts, QA team and Developers • Multilingual Test Scenarios (35+ Languages) • Right tools and the right approach • Remote and Cloud Execution • Parallel Executions • Framework and Programming scalability • Continuous Integration Key takeaways
- 21. 21 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved. • IDC : IDC May 2015 • Jigesh Shah & Mradul Kapoor : “Traceability and Automation using BDD” • Cucumber : https ://cucumber.io/ • Parallel and Web Automation : http ://www.seleniumhq.org/ • Mobile Automation : http ://appium.io/ References
- 22. Thank You!