The Best of Both Worlds - Combining Performance and Functional Mobile App Testing
Download as PPTX, PDF0 likes990 views
The document discusses the importance of combining performance and functional testing in mobile app development, especially during peak seasons like Black Friday. It highlights statistics on mobile shopping trends and the critical impact of site load times on user conversion rates. Additionally, it emphasizes strategies for proactive monitoring and testing to ensure optimal user experience and performance across varying mobile platforms.
The Best of Both Worlds - Combining Performance and Functional Mobile App Testing
- 1. The Best of Both Worlds - Combining Performance and Functional Mobile App Testing
- 3. Presenters Speaker Henrik Rexed Performance Engineer Neotys Speaker Lingkai Shao Marketing Analyst Bitbar Speaker Dave Horner Director of Sales Bitbar
- 4. Neotys Continuous Performance Testing Solution System Testing Integrated Component Testing/System-wide Testing ReleaseComponent Testing Continuous Integration & Component Testing Proactive Production Monitoring End-User Experience
- 5. “Continuously ship quality apps with a high velocity delivery pipeline” Bitbar - Mobile DevOps
- 6. Dev Ops Continuous Release & Deployment Collaborative Development Continuous Testing Continuous Monitoring Continuous Integration Continuous Feedback DevOps is no longer a unicorn’s discipline. TestDev Ops Shift Left Shift Right DevOps
- 7. According to Adobe 2016 Online Shopping Data, mobile drove 55% of visits to retail websites on Black Friday (45% coming from smartphones, 10% from tablets) and accounted for 36% of sales (25% from smartphones, 11% from tablets) * Why Functional and Performance Testing for Holiday Season *=http://news.adobe.com/press-release/marketing-cloud/media-alert-adobe-data-shows-black-friday-breaks-online-sales-record-3
- 8. Performance matters 53% of visits to mobile sites are abandoned after 3 seconds according to research from Google According to Google doubleClick faster websites has 70% longer average session lengths, 35% lower bounce rates and 25% higher ad viewability than their slower counterparts. Etam reduces its average page load time from 1.2s to 500ms and increased conversions by 20% Walmart saw up to a 2% increase in conversions for every 1 second of improvement in load time. Every 100ms improvementalso resulted in up to a 1% increase in revenue.
- 9. Top Failures During Peak Days
- 10. Concrete study
- 11. When Revenue is Generated
- 12. ● Preparation: typically 10 weeks ahead of peak days ● Functional test suite including Stage One: Functional & Compatibility
- 13. Do you know the behavior of your app/web users? Defining the right user flows
- 14. Stage One: Functional & Compatibility
- 15. Stage One: Functional & Compatibility
- 16. Stage Two: Performance & Load
- 17. Standard performance testing workflow Build the performance strategy Create the testing scenarios Prepare Execute Analyse Tuning
- 18. Performance design is the key
- 19. Mobile Application Architecture Landscape Native app Hybrid app Web app Web app Web Model Mobile Model Rest Interface Business logic Data Access logic XML Legacy Relational Presentation Business Data Third party
- 20. Mobile Performance Testing Touchpoints BACK-END EXECUTION PLATFORM RIA/FRAMEWORKS DATABAS E NETWORK 3G, 4G / LTE, Wi-Fi VIDEO HTTP Adaptive StreamingThroughput Latency Packet loss SERVER INFRASTRUCTU RE OS CLIENT APP DEVICE iPhone 4, 5, 6 Samsung Blackberry … (7,000 different device types are used to access Facebook) Screen Size CPU Memory Native Browser-Based Hybrid
- 21. Impact of Mobile Network
- 22. The most important network constraints are: • Latency • Bandwidth At a certain point, increasing the bandwidth will only have a minor effect. Latency & Bandwidth . . .
- 23. Packet Loss Consequences • Packet Retransmission • End user experience degraded because packet loss increases jitter • Errors could occur if packet loss is in pure data transmission Packet Loss is Inevitable with Mobile • Mobility: Quickly switching to the next cell tower may result in temporary signal loss due to congestion on the current tower Mobile Protocols Try to Reduce Packet Loss Rate Packet Loss . . .
- 24. Lots of users • Mobile applications often have large user bases • High growth in popularity can create the need for highly scalable load generation infrastructure Geographical distributed load • Users are accessing apps from locations around the globe • Distance and location can greatly affect performance Mix of Web & Mobile Users • Many mobile apps have a corresponding web app with a shared backend • High load on the web app can affect performance on mobile High volume, Geographically Distributed
- 25. Don’t forget the User Experience….. What are is really being measured?
- 26. … Applying the Bitbar Testing Integration… 00:00:00 00:00:30 00:01:00 00:01:30 Load NeoLoad Application Server App Error Rate Virtual Users Load User Experience
- 27. RUM (Passive) vs Synthetic (Active) Stage Three: Proactive Production Monitoring
- 28. Why Synthetic Monitoring? ● Alarm and Alert ● Better MTTR! (Mean Time To Resolution) ● Find and fix issues before impact to users ● Baselining and Benchmarking (Pre-production Staging & Post Production) ● Prepare for peak traffic and new markets ● Monitor Complex Transactions ● Measure and Adhere to SLAs ● Hold 3rd Party Vendors Accountable ● Test from end user perspective ● Competitive Benchmarking Stage Three: Proactive Production Monitoring
- 29. Understand Performance of Complex User Transactions and Hold 3rd Party Services Accountable (SLAs) Benefits of Synthetic Monitoring
- 30. Faster Mean Time to Repair… Identify Issues Before Your Users Do Benefits of Synthetic Monitoring
- 31. Understand Performance In New Markets, Optimize Content Delivery, Benchmark Pre & Post Change Benefits of Synthetic Monitoring
- 32. Competitive Benchmarking: Benefits of Synthetic Monitoring
- 33. Purpose Built Synthetic Monitoring Solution
- 34. Global Points of Presence
- 35. Use Your Existing Functional Test Scripts For Monitoring
- 36. Solution: Stage Three: Proactive Production Monitoring
- 38. Questions? A Q