Techniques, Tips & Tools For Mobile App Testing
Download as PPTX, PDF1 like908 views
The document discusses strategies for improving mobile test automation, emphasizing the importance of real-time feedback and continuous integration to enhance app quality and delivery speed. Key techniques include utilizing small batch sizes, comprehensive version control, and developer-tester collaboration, which have significantly reduced testing times. Additionally, it highlights the value of leveraging real user data and emerging technologies for effectively driving development and testing processes.
Techniques, Tips & Tools For Mobile App Testing
- 1. © 2015 SOASTA. All rights reserved. June 26, 2015 1 Brad Johnson VP Product Marketing @bradjohnsonsv Vancouver, BC June 24, 2015
- 2. TIME
- 3. EXPERTISE
- 4. PROCESS
- 5. Says who
- 6. you.
- 7. 250 Testers from 6 Continents
- 8. Mobile is your chance to make a CHANGE
- 10. June 26, 2015 10© 2015 SOASTA. All rights reserved. Seek TIPS from experts
- 13. REALITY always trumps speculation Static aggregates (ok)
- 14. REALITY always trumps speculation Real Time (best)
- 15. REALITY always trumps speculation Data science - better than best!
- 16. June 26, 2015 16© 2015 SOASTA. All rights reserved. ASAP Simple Time Consuming Mundane Rhythm Goal : Continuous Integration execution
- 17. June 26, 2015 17© 2015 SOASTA. All rights reserved.
- 18. June 26, 2015 18© 2015 SOASTA. All rights reserved. o Small Batch Sizes
- 19. June 26, 2015 19© 2015 SOASTA. All rights reserved. o Small Batch Sizes o Comprehensive Version Control
- 20. June 26, 2015 20© 2015 SOASTA. All rights reserved. o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies
- 21. June 26, 2015 21© 2015 SOASTA. All rights reserved. o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation
- 22. June 26, 2015 22© 2015 SOASTA. All rights reserved. o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback
- 23. June 26, 2015 23© 2015 SOASTA. All rights reserved. o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback o Emphasis on working builds
- 24. June 26, 2015 24© 2015 SOASTA. All rights reserved. o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback o Emphasis on working builds o Consistent environments
- 25. June 26, 2015 25© 2015 SOASTA. All rights reserved. o Small Batch Sizes o Comprehensive Version Control o Simple Branch Strategies o Automation o Continuous Feedback o Emphasis on working builds o Consistent environments o Developer – Tester Collaboration
- 26. June 26, 2015 26© 2015 SOASTA. All rights reserved. Device reaction to app functionality Battery, CPU, Memory Dependent services and 3rd party apps/ads Separate back and front end performance User experience under every traffic condition Performance to revenue correlation
- 27. June 26, 2015 27© 2015 SOASTA. All rights reserved. Seek TIPS from experts
- 28. June 26, 2015 28© 2015 SOASTA. All rights reserved.
- 29. June 26, 2015 29© 2015 SOASTA. All rights reserved. RetestTest Development Infra setup RepeatFixTest Analyze TIME Beware Automation Solutions that Hinder Speed Developers code tests, but not faster.
- 30. June 26, 2015 30© 2015 SOASTA. All rights reserved. RetestTest Development Infra setup RepeatFixTest Analyze TIME Analysis is still slow & manual Beware Automation Solutions that Hinder Speed
- 31. June 26, 2015 31© 2015 SOASTA. All rights reserved. RetestTest Development Infra setup RepeatAnalyze Develop plans Post-test Validate Traditional Approach Test Fix / Analyze Fix Test Dev Test / Analyze Fix Test Dev Test / Analyze Fix Test Dev Test / Analyze Fix Test Dev TestModern Approach OR / Analyze Fix Test Dev Test / Analyze Fix Test Dev Test AUT/device setup Manual
- 32. June 26, 2015 32© 2015 SOASTA. All rights reserved. 250 hours of manual tests compressed to 1 hour Regression tests not run until feature complete now run nightly 4 months to overnight 25% faster Time To Market
- 33. June 26, 2015 33© 2015 SOASTA. All rights reserved. “(Mobile test automation) has been critical in improving both the speed to deliver and the quality of our mobile apps.” “We decreased our test cycles from three weeks to a couple of days.” 3 WEEKS 2 Days
- 35. RUM = “What to Test” (real user monitoring)
- 36. June 26, 2015 36© 2015 SOASTA. All rights reserved. o Rapid & accurate test development o Reliable, maintainable test automation o Stability across mobile OS platforms o Device performance measurement o Real-Time feedback (via Continuous Integration is a +)
- 38. June 26, 2015 38© 2015 SOASTA. All rights reserved. Rank active devices Device usage Weight matrix to device Know problem areas per device Device Access Private Public
- 39. June 26, 2015 39© 2015 SOASTA. All rights reserved.
- 40. June 26, 2015 40© 2015 SOASTA. All rights reserved. Source: Mobile Test Coverage Index (Perfecto)
- 41. June 26, 2015 41© 2015 SOASTA. All rights reserved. o Behind the firewall o Always available o Online access for all your teams o 24x7, secure, remote access to devices o Manual & automated testing Hands on when you need them.
- 42. June 26, 2015 42© 2015 SOASTA. All rights reserved. o Dedicated o Latest in-market devices o Maintained for you o Secure o API’s to install apps, reboot/wake-up, launch tests Perfect for automated mobile regression testing
- 43. June 26, 2015 43© 2015 SOASTA. All rights reserved. Analytics Database Main SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Database CDN Credit Card Web analytics Other Svcs… Performance Tester(s) Virtualized services3rd party services Load Testing Tool
- 44. June 26, 2015 44© 2015 SOASTA. All rights reserved. Analytics Database Main Analytics Load GeneratorLoad GeneratorLoad Generator SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Database CDN Credit Card Web analytics Other Svcs… Performance Tester(s) Virtualized services3rd party services Cloud Servers Load Testing Tool
- 45. June 26, 2015 45© 2015 SOASTA. All rights reserved. Analytics Database Main Analytics Load GeneratorLoad GeneratorLoad Generator SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Database CDN Credit Card Web analytics Other Svcs… Performance Tester(s) Virtualized services3rd party services Cloud Servers Load Testing Tool
- 46. June 26, 2015 46© 2015 SOASTA. All rights reserved. Analytics Database Main SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Database Analytics Load GeneratorLoad GeneratorLoad Generator CDN Credit Card Web analytics Other Svcs… Performance Tester(s) Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. 3rd party services Virtualized services Cloud Servers Real time analytics Load Testing Tool
- 47. June 26, 2015 47© 2015 SOASTA. All rights reserved. Utilize Real User data to drive dev & test specifications
- 48. June 26, 2015 48© 2015 SOASTA. All rights reserved. CI runs unit tests Utilize Real User data to drive dev & test specifications
- 49. June 26, 2015 49© 2015 SOASTA. All rights reserved. CI runs unit tests Utilize Real User data to drive dev & test specifications Spin up suitable, consistent test environments in the cloud or the lab
- 50. June 26, 2015 50© 2015 SOASTA. All rights reserved. Mobile automation runs in remote or on premise devices CI runs unit tests Utilize Real User data to drive dev & test specifications Spin up suitable, consistent test environments in the cloud or the lab
- 51. June 26, 2015 51© 2015 SOASTA. All rights reserved. Iterative load tests run as CI tasks Mobile automation runs in remote or on premise devices CI runs unit tests Utilize Real User data to drive dev & test specifications Spin up suitable, consistent test environments in the cloud or the lab
- 52. June 26, 2015 52© 2015 SOASTA. All rights reserved. Iterative load tests run as CI tasks Mobile automation runs in remote or on premise devices CI runs unit tests Daily (or more frequent) updates keep teams quality focused Embedded failure details & bottleneck data isolate issues Utilize Real User data to drive dev & test specifications Spin up suitable, consistent test environments in the cloud or the lab
- 53. June 26, 2015 53© 2015 SOASTA. All rights reserved. Iterative load tests run as CI tasks Mobile automation runs in remote or on premise devices CI runs unit tests Manage to a performance baseline Daily (or more frequent) updates keep teams quality focused Embedded failure details & bottleneck data isolate issues Utilize Real User data to drive dev & test specifications Spin up suitable, consistent test environments in the cloud or the lab
- 54. June 26, 2015 54© 2015 SOASTA. All rights reserved. EXPERTISE
- 55. June 26, 2015 55© 2015 SOASTA. All rights reserved. TIM E
- 56. June 26, 2015 56© 2015 SOASTA. All rights reserved. PROCESS
- 57. June 26, 2015 57© 2015 SOASTA. All rights reserved. Mobile Test Automation
- 58. June 26, 2015 58© 2015 SOASTA. All rights reserved. Mobile Test Automation Real Test Devices
- 59. June 26, 2015 59© 2015 SOASTA. All rights reserved. Mobile Test Automation Performance Testing Real Test Devices
- 60. June 26, 2015 60© 2015 SOASTA. All rights reserved. Mobile Test Automation Performance Testing Real Test Devices Solid Integration
- 61. Mobile is your chance to make a CHANGE
- 62. You CAN do it!
- 63. June 26, 2015 63© 2015 SOASTA. All rights reserved. SOASTA.com o Whitepaper: 7 Steps to Pragmatic Mobile Testing • http://info.soasta.com/Whitepaper-7-Steps-to-Pragmatic-Mobile-Testing.html o July 7 Webinar “Strategies for Mobile Automation Success” • http://info.soasta.com/strategies-for-mobile-test-automation-success.html o Recorded webinars - e.g. “Pragmatic Mobile Testing” • http://www.soasta.com/webinars/ Others o “Future of Testing” (search Stickyminds/Techwell) o “Mobile Test Coverage Index” (search Perfecto) See “7 Steps” Session @ 3:00pm!
- 64. June 26, 2015 64© 2015 SOASTA. All rights reserved. @bradjohnsonsv bjohnson@SOASTA Brad Johnson
Editor's Notes
- #17: 5. KNOW HOW TO AUTOMATE Earlier in this paper, we discussed the importance of analyzing and understanding the testing matrix. This helps you prioritize, for instance, core functions vs. edge functions. That analysis also plays a crucial role in helping us determine which tests are best suited for automated testing approaches. There are a few tricks to determining how to automate (the first four strategies in this paper prepare you for this assessment). One of the keys may not be intuitive: timing. With testing automation, it is essential to get started as quickly as possible. Delays can create backlogs that become impossible to overcome. Ideally, we want to automate the test cases that are repetitive. Key experiences that are in your regression suite are the best targets. These are the tests that you want to repeat every time a build is complete to ensure nothing has been inadvertently broken in/by the new build. The next factor is evaluate how long your tests take. Time-consuming, but simple, tests are the hardest for humans to repeatedly execute. That’s where most human error occur, due to fatigue. But that’s also where an automated testing solution can quickly free up resources to allocate to more complex testing tasks. It might be tempting to try to automate the most challenging components of the test plan, but in actuality, we want to take the opposite tack. Save the hard tests for your people, and use those tests to challenge your engineering/testing teams. Build your regression suite of tests with the simple and most repeatable tests, and your team productivity will increase significantly and consistently with every iteration.