Cross platform test automation using Appium

Editor's Notes

• #3: Good evening all Before I begin I would like to thank Ady, Vikas and everyone else involved in organising the meetup. You guys are doing a brilliant job. About me – I work as a QA lead for the mobile team at MYOB since last 1 year and was on their flagship windows product prior to that. Before joining MYOB I was working with TW Having worked for these companies has given me opportunities to work on a variety of platforms and tech stacks but I sincerely believe there is still heaps to learn I love travelling and exploring new places.
• #4: So that was a bit about me, now I would like to know a bit about you all. Who here has worked on mobile project Have you used some sort of automation tool Great, so we do have a good mix here – people who have worked on some of these areas and some who are keen to learn more
• #5: So what’s the big deal about mobile
• #6: A recent survey by Gartner indicates that by 2018 more than 50% of the users will use a mobile device first in order to access the internet. Well I would say mobile is not the future anymore in fact it is the present now.
• #7: Another study by Morgan Stanley suggests that early last year the number of mobile users have already surpassed the total number of desktop users globally. Therefore, as a result - a lot of companies and products are now moving to the mobile first strategy. They want to make sure whatever you are able to do on their web site you should be able to do on your mobile too. I don’t remember when was the last time I logged in to facebook on my laptop, neither have I logged in to my bank’s website for quite some time
• #8: SO far I have used the word “apps” quite a few times. Now let us understand what does an app mean? Primarily there are 3 kinds of mobile apps – Native, Web and Hybrid Native - A native app is an app developed specifically for the targeted phones operating system, for example iOS or Android. The user experience, interaction with the phones technology & performance of these apps is unrivaled. There is a considerable dev and testing effort involved because you are dealing with 2 entirely different codebases. Key words you may hear when explaining the development environment are Objective-C, Swift, Java. This app is available in the app store. Responsive – Mobile web or often referred to as responsive is nothing but a web page that renders to a wide range of devices. Imagine opening Google or yahoo on your phone’s browser and you will notice that despite so much of content and such small screen of your mobile the content doesn’t get distorted. So these are not available on app store and rather you simply open them on your browser. Hybrid - A hybrid app is a mix between web and native. Technologies such as Phonegap and Cordova or Titanium allows developers to take web technology and wrap it in a native shell. This is done so the app can take advantage of the phones hardware and features but can also be ported across platform. These are again downloadable from the app store so from a normal user’s perspective they don’t see much of a difference.
• #9: Talking about our apps – we have 2 apps – PayDirect and OnTheGo which are available for both android and iOS These are all native apps. So technically there are 4 apps to be taken care of.
• #10: PayDirect allows our customers to pair the card reader via bluetooth and then take credit card payments. They can swipe, insert or tap the card in order to do so. OnTheGo is primarily meant for raising invoices. Its being used by a lot of tradees or small businesses who have to visit different places. So imagine a plumber coming to your place who aftre doing his job can straight away raise an invoice and get his money right then. Its gaining a lot of traction in our small and medium enterprise business division. Apart from creating invoices, you can also create contacts or get in touch with our customer care. There is a but of feature overlap in these invoices. We are still adding more features and typically do a release every 2 weeks. So you can imagine we always have to be on our toes. 15% of SME businesses are using these apps now and growing 10% compounded monthly
• #11: Big challenges on small devices. Mobile apps pose new challenges to testing: Rating and Reviews. Bad quality may cause trouble to the reputation of the company and brand. In the era of millions of charts, lists and reviews there hasn’t been a case when a company released a low-quality app and people didin’t find out about it. The influence of bad reviews can be drastic. Device and OS distribution. Mobile devices change and improve with unprecedented speed. Annually hundreds on new devices appear on the market. Different screen sizes, form-factors and the variety of OS often cause problems to apps. Urgency. You need to be fast, extremely fast to move with the times. Teams should create design, write code, test and release an application within a very short period of time. Traditional manual testing doesn’t meet the speed requirements of mobile world. Complexity. All the mobile devices are becoming more and more elaborate, and are based on more and more advanced technologies. Location recognition, Wi-Fi, real-time events, popups – all this makes both manual and automated testing way more complicated.
• #12: I would say automation could be considered both challenge and a solution for all these other challenges. Because the time to market is so short you cant really rely on manually testing all the flows in the app when doing a release. So its best to automate your test cases. But please while doing so be congnizant of the basics such as testing pyramid.
• #13: There are a plenty of automation tools available in market. I have put a few of those here on this slide. Some of these are platform specific whereas others support cross platform automation. Robotium is an Android test automation framework that fully supports native and hybrid applications. Robotium makes it easy to write powerful and robust automatic black-box UI tests for Android applications. With the support of Robotium, test case developers can write function, system and user acceptance test scenarios, spanning multiple Android activities. UIautomator, by Google, provides an efficient way to test UIs. It creates automated functional test cases that can be executed against apps on real Android devices and emulators. It includes a viewer, which is a GUI tool to scan and analyze the UI components of an Android app. Espresso, by Google, is a pretty new test automation framework that got open-sourced just last year, making it available for developers and testers to hammer out their UIs. Espresso has an API that is small, predictable, easy to learn and built on top of the Android instrumentation framework. You can quickly write concise and reliable Android UI tests with it. Calabash is a cross-platform test automation framework for Android and iOS native and hybrid applications. Calabash’s easy-to-understand syntax enables even non-technical people to create and execute automated acceptance tests for apps on both of these mobile platforms.
• #14: Not very long ago our test stack used to look something like this. For iOS we were using UIAutomation for test automation. For android were had all our tests automated using Robotium We were also using Kiwi and KIF for Unit and Integration testing in iOS For Android unit testing we use Robospock Apart from the apps we also have a server component which handles most of the business logic. The server is ruby code base and we use cucumber for automating server acceptance tests So even though from business perspective these are more or less identical apps, we were compelled to maintain 2 entirely different test suites.
• #15: So we knew there was scope of improvement and we started jotting down what were the things we were looking for
• #16: First and foremost we wanted to use a cross platform automation tool that could allow us to run the same test against both android and ios. This will mean Reusable Code And Less maintenance effort
• #17: While as of now all our apps are native, our tech roadmap is quite diverse and there is a potential that we may create hyrbrid or web apps in future. So wanted to take our tech roadmap into consideration and invest in a tool that could automate these other types as well.
• #18: As new devices and operating system versions are released every day we want to go for a tool which is up to date with all these developments. And also the one which could allow to increase our coverage as and when required.
• #19: As a next step to scalability we wanted to look for a tool which is compatible with a cloud based testing service provider. There are cloud based testing service providers that allow you to run your tests on an array of devices and operating systems and then generate a detailed report.
• #20: So far we have had our test written in tools which only supported a specific scripting language. Which means if a test fails or there is any issue only the people who have worked on that language will be able to investigate. We wanted a tool that could allow us to chose a language of our choice.
• #21: We definitely wanted to add BDD layer to our tests. So that these tests could be written even by the non technical stake holders of the project such as product owner. Also the BDD tests act as a living documentation.
• #22: Keeping all those motivations in mind, we started looking for cross platform automation tools and narrowed down to these. MonketTalk lost early in the race because of various reasons. In fact it has been acquired by Orcale recently so its future remains uncertain. Calabash is definitely good and I would say as popular as Appium The github stats vary slightly. However, Appium’s USP is that it doesn’t require you to alter the app because the appium server utilizes the platform specific tools in order to gain control of your device. I will talk about its architecture in a bit more detail soon. Calabash is undergoing a revamp and will come up as 2.0 wherein they are re-writing the API from scratch so there could be some major changes coming up
• #23: So why did we chose appium Supports both Android and iOS Because it takes advantage of selenium’s json wire protocol, all the languages that can be used to write selenium tests can be used to write appium tests too. Supports native, web and hybrid It allows you to run your tests in an emulator or on a real device Being selenium based, it follows a well formulated API so if you have ever worked on selenium before, picking up Appium becomes quite easy. Appium uses 2 technologies: the Selenium API (that is, a subset of Selenium commands that are relevant for mobile testing), and the WebDriver JSON Wire Protocol. The Selenium/WebDriver combo is a de-facto standard in end-to-end system testing and is well-known to many developers and testers.
• #24: Appium is sponsored by Saucelabs and therefore it seemlessly integrates with their cloud offering. However, it is compatible with most of the other players in the market. It does support BDD which was another key motivation for us. There is a strong community support. They have their own discussion forum and even the issues raised on github get responded to pretty quickly. As I said before, it allows you to Test the same app that you would submit to app store – you do not need to compile a third party framework into your app’s code that needs to be stripped out later. Avoids the risk of introducing any unforeseen bugs. Test the same app as you would release. And last but not the least, it is open source so its free and you can fork and create your own version if you want.
• #25: So how does Appium do all this? Let’s try to understand what happens behind the scenes. Appium basically is an HTTP server written using nodeJS that exposes a REST API Provides common front end API to the totally different back ends It is almost the same as the Selenium server that gets HTTP requests from Selenium client libraries. Step 1 – commands are sent in form of JSON via HTTP Step 2 – Appium invokes platform specific tools to execute the commands Step 3 – Client sends back response to the appium server Which is then logged on the console When I say platform specific Appium uses UIAutomator which is capable of controlling the entire device on Android and execute these commands Similarly, for iOS it uses Instruments which is again provided by apple and is capable of controlling and executing commands on an ios device At the webscripts level, it uses desiredcapabilities the same way you can do it for dowing cross browser automation with selenium.
• #26: Appart from running the appium server from the command line, you can also launch it via UI. The Appium app is capable of doing pretty much everything that you may want to do with command line, so you don’t need to worry about Node. Clicking on the search icon here takes you to the appium inspector
• #27: The inspector enables you to check out the hierarchy of your app. This can come in handy when writing tests. equivalent of selenium IDE Also supports basic record and playback
• #28: We also added a BDD layer to our appium tests, in order to do this we were using cucumber to write scenarios The step definitions were written using Ruby And we are following page object pattern. The Page Object pattern represents the screens of your app as a series of objects. You can separate out your testing code from the app screens. It also provides you the ability to reuse a screen in multiple tests. So if there is a UI change required, you don’t need to modify all the tests rather just update the page (screen)
• #29: Here is a small example of a test scenario written using cucumber All I am trying to do here is Launch the app Login using valid credentials Once I successfully login, I should be taken to the setup passcode screen
• #30: Here is the step definition for this scenario written in ruby As you can see we have tried to write minimal code here and rather are calling our screens whenever we want to do a particular operation on those screens
• #31: Here is how a sample screen would look like They are both inheriting a common base class They are a few differences in these screens The title for Android screen is MYOB where as for iOS it is login Similarly, we are using IDs for android and name for iOS It’s a good idea to keep pages separate for both the platforms so that change in one app doesn’t impact the tests for the other platform
• #32: Now I thought because login is such a quick scenario it will hardly take a few seconds to execute and so I have chosen a slightly longer feature for the purpose of demo So in this example here I am testing the dashboard feature GO TO NEXT SLIDE ======================= So as you can see as part of these scenarios I drill into various areas of the app and assert that I am actually there GO TO DEMO
• #33: So after all these recent changes, our test stack has grown up a bit. We have this another layer of acceptance tests written using Appium and Cucumber
• #34: Appium is No silver bullet. There are some limitations. For example – you can only run 1 iOS test at a time. This is also because of the fact that Appium uses apple’s instruments framework under the hood and you can have only one instance of instruments running at a time. For android you need to start multiple appium servers with different flags It has limited support for Android versions lower than 4.1. There is a work around available by using Selendroid. Appium extends the WebDriver spec with extra commands to support mobile gestures such as long tap and shake. However, not all mobile gestures are supported out of the box and you may have to write your own methods in order to make some gestures work. Some people find this annoying. As of now it supports Android, iOS and even Firfox OS but doesn’t support Windows Mobile Appium doesn’t have the ability to do image comparison. So if you are after something like that, tool like Sikuli may better suit your needs. Fortunately, none of these limitations have been showstopper for us. But its good to be aware of both pros and cons before zeroing down on a particular tool.
• #35: As I said we have just started with Appium for last few months and there is still a long road ahead of us in order to achieve NIRAVANA 1st of all we need to migrate all our existing tests to Appium. Because we have a large number of tests written in the other tools which are absolutely valuable till we port them all into appium, we nee to have both the test frameworks in our pipelines. We are making sure all new tests are being written in Appium. Also, as and when we add new tests to appium we take a momemnt to review the existing tests to see if something got covered and we can get rid of those tests. We have been running our tests on simulators and emulators in the pipelines. However, its always best to run your tests on a real device. Appium does support that. In order to achieve this in a scalable manner we are evaluating cloud service offerings to see which one suits us the best. So there are more milestones for us to achieve but Appium certainly looks promising and seems to be the step in right direction.
• #36: I hope I ddnt bore you enough, feel free to ask questions now or get in touch via linkedin and I will be happy to help Q. Can you test web view applications with appiu? A. Yes you can. Tests can be run in context of a webview, pretty much execute selenium tests on the mobile web view Q. How scripts handle crashes? A. Scripts handle crashes by ending the session and you get an error code