![[HCMC STC Jan 2015] FATS: A Framework For Automated Testing Scenarios](https://cdn.slidesharecdn.com/ss_thumbnails/thoquanaframeworkforautomaticconstructionoftestscenariosformuse-cases-150209012507-conversion-gate02-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (13)
Ad
Similar to An Automation Framework That Really Works (20)
Ad
An Automation Framework That Really Works
- 1. An Automation Framework That Really Works By Basivi Reddy Junna QA Practice Manager and Sr. Automation Architect Thursday, May 3,2012
- 2. An Automation Framework That Really Works - Basivi Reddy Junna 2 Almost anything that you would ever want to do in programming can be done with Functions
- 3. An Automation Framework That Really Works - Basivi Reddy Junna 3 Traditional approach to automation Why automation frameworks at all? Disadvantages of automation frameworks Available frameworks of automation Data Driven Framework Keyword Driven framework Modular Framework Functional Decomposition Framework • Conceptual overview • Components • Workflow • Driver Hybrid Framework Q&A Introduction
- 4. 4 Create Tests •Record the test or Program it manually Enhance Tests •Synchronization •Parameterization •Checkpoints Execute Tests •One by one •Batch mode Analyze and Report •Analyze Results •Report and Manage Defects Traditional Approach to Automation An Automation Framework That Really Works - Basivi Reddy Junna Create Enhance Execute Analyze and Report
- 5. 5 Traditional Approach to Automation Continued… An Automation Framework That Really Works - Basivi Reddy Junna Pros • Best approach for automating few tests • Quick and easy to implement Cons • In the absence of any common pattern or structure, reusability of automation code is very limited • Standardization is a challenge • Difficult to maintain as tests are standalone and each test is a separate point of maintenance • As the number of tests increase, coverage analysis becomes difficult and test suite may quickly go out of control • Difficult to consolidate and analyze results
- 6. 6 Standard wish list expected to be fulfilled by automation An Automation Framework That Really Works - Basivi Reddy Junna Test Development Promote reusability of assets Provide rapid test building capabilities Even manual testers should be able to create automated tests Test Data Management Test data management should be a breeze. Data coverage analysis should be real simple Use the data generated by one test in others Handle non-reusable sets of test data without manual intervention Test Execution Execution setup should be centralized. Even non technical people should be able to setup and run automated tests. Run tests in multiple environments concurrently Run a batch of tests with just click of a button Schedule tests to run over night Recover from any errors and continue with other tests in the batch Re-run tests that did not run completely in previous run. Re-run them from the point where they left off. Re-run tests that have one or more failed checkpoints or assertions in previous run Test Results and Reporting Notify designated people when tests fail to execute Create summarized and detailed reports with results of execution. Keep history of results. Compare results of two or more test runs Maintenance Fewer points of maintenance Quickly identify maintenance points when application changes Maintenance should be super simple
- 7. 7 Why Use Automation Frameworks At All? An Automation Framework That Really Works - Basivi Reddy Junna Robust Do not break easily. Recover from errors for un- interrupted execution Centralized Execution Control and unattended execution. Scalable Can easily be up-scaled in terms of number and size of tests and yet retain control Flexible With little modifications assets can be leveraged to create new tests and cover more scenarios Maintainable Easy to maintain - Few maintenance points due code re- use Reusable Assets are reusable and provide rapid test building capabilities
- 8. 8 Difficult to implement Availability of skilled resources is an issue Require more up front investment than traditional approach Over engineering trap Wrong selection of framework Get obsessed with framework Disadvantages of Automation Frameworks An Automation Framework That Really Works - Basivi Reddy Junna
- 9. 9 Data Driven Framework Key word Driven Framework Modular Framework Functional Decomposition Framework Hybrid Framework Available Frameworks of Automation An Automation Framework That Really Works - Basivi Reddy Junna
- 10. 10 Data Driven Tests: • Parameterized tests are commonly referred to as data driven tests • Parameterization: Fixed input values for test steps are replaced with variables and variables are supplied with values during runtime • Same steps and logic are repeated across multiple test data sets Data Driven Framework: • Will have all features of data driven tests • Logic is controlled by test data. In other words script’s control flow and other behavior is dictated by test data • Suitable for large data intensive applications with limited variations in functional flows and simple to moderately complex business rules • Usually made up of few but very large test scripts whose behavior is controlled by data populated in test data repository • Test maintenance is mainly taken care by updating test data Data Driven Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 11. 11 Data Driven Framework - Example An Automation Framework That Really Works - Basivi Reddy Junna Test Id TestProd_Id TestAddress_Id TestPmt_Id 1 2 1 2 2 3 2 1 Id Product_Id Action 1 10 Search 2 3 Add 2 4 Remove Tests Test_Products Id Product_Name 1 Kindle Fire 2 iPod Charger 3 Google Nexus 4 Blackberry Products Id Address_Id Action 1 3 Add 2 1 Edit Test_Addresses Id Address City 1 … Chicago 2 … Chicago 3 70 York… Lombard Addresses Input Test Id
- 12. 12 Pros • Limited numbers of tests to maintain • Script maintenance is largely handled through test data updates Cons • Complex to implement: Test data repository may become as complex as the application data model itself • Not suitable for applications with heavy variations in functionalities and flows • Tests scripts are too big to fail Data Driven Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 13. 13 Testing activities can be divided into basic actions called keywords – Click, Select, Enter etc Keyword(s) => Function(s) Test scripts are written using key words Keywords (functions) are invoked by providing as inputs - data and application objects Automated scripts are automatically generated behind the scene, while you are creating manual scripts using keywords Keyword Driven Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 14. 14 Pros: • You create manual tests and have automated scripts generated behind the scene automatically • Even manual testers can create automated tests without any knowledge or experience of automation. • Two birds with one shot - Produces highly structured manual tests so you can test manually in case if automation is NOT working for any reason Cons: • Debugging is very tedious • Heavy upfront investment • Suitable for small and medium applications only • Difficult to handle dynamic objects • Grouping steps and controlling iterations is extremely difficult • Complex test flows cannot be specified using keywords Keyword Driven Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 15. 15 Divide the applications into logically identifiable sections called modules Develop automation scripts for the modules Modules characteristics: • Re-usable • Data Driven – Modules behave according to data • Perform multiple functions and validations • Handle errors • Report results Actual tests are created by calling the reusable modules in a required sequence Can be treated as an additional layer of abstraction on the top of functional decompositions framework Modular Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 16. 16 Pros: •Large number of tests are created by relatively less number of modules. Hence, fewer points of maintenance •Provides much higher level of abstraction than Keyword driven or functional decomposition frameworks making it easy for users to understand the test flows. Cons: •Best described as half data driven and half functional decomposition framework. It has disadvantages of both the frameworks but not the advantages of Functional Decomposition Framework •Restricted maneuverability - Not suitable for application with wide variety of functionalities, twists and turns(flows). Integration of modules into actual tests and still conforming to navigational rules of the application is difficult to achieve •Each module performs multiple functions. Selective execution of functions is difficult to achieve •Combination of higher level of abstraction and underlying data driven framework make it difficult to understand the test data being passed to and consumed by tests Modular Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 17. 17 Divide the application into logically identifiable functions Create functions using scripting language of the tool Make the functions data driven using parameters Create actual tests using the functions Suitable for medium to highly complex applications Can handle wide variety of functionalities Capable of being data driven Functional Decomposition Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 18. 18 Functional Decomposition Framework: Example Test 1 An Automation Framework That Really Works - Basivi Reddy Junna Launch Application Login Search Products Add To Cart Edit Cart Checkout Shipping Address Payment Method Place Order Logout Test 1Launch Application Login Search Products Add To Cart Edit Cart Checkout Shipping Address Payment Method Place Order Logout Launch Application Login Search Products Add To Cart Edit Cart Checkout Select Shipping Address Select Payment Method Place Order Logout URL UID and PWD Criteria & Product Info
- 19. 19 Functional Decomposition Framework: Example Test 2 An Automation Framework That Really Works - Basivi Reddy Junna Launch Application Login Search Products Add To Cart Edit Cart Checkout Shipping Address Payment Method Place Order Logout Test 2Launch Application Login Search Products Add To Cart Edit Cart Checkout Shipping Address Payment Method Place Order Logout Launch Application Login Search Products Add To CartEdit Cart Checkout Select Shipping Address Select Payment Method Place Order Logout URL UID and PWD Criteria & Product Info Quantity Capture Order Number
- 20. 20 Test Data Repository Business Functions Common Functions Generic Functions Framework Functions Functions for Web Applications Functions JAVA Applications Test1 Function 1 Function 2 … Function n Test2 Function 1 Function 5 … Function n Test3 Function 6 Function 10 … Function n Driver Test1 Test2 Test3 … Test n Functional Decomposition Framework – Conceptual Overview Functions for Mainframe Applications Etc…
- 21. 21 Business Functions: These are re-usable functions that perform specific tasks by reading data from test data repository. All functions are data driven. These functions are specific to the application under test. Using common functions (explained later) will significantly accelerate the development of business functions Characteristics: • Validate the inputs • Verify execution status of previous functions in the test • Verify start status • Implement synchronization and perform validations • Recover from errors • Report results Components of the Functional Decomposition Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 22. 22 Common Functions: These functions are independent of any business domain/application and are commonly used. There are three sub categories of common functions: 1. Generic functions (string functions, date functions etc) 2. Framework functions (Reporting, configuration, error handling functions etc) 3. Environment specific functions such web functions, Power builder functions etc. These functions are mostly called by business functions Automated Tests: Automated Tests are created by simply calling a series of re-usable business functions in required sequence using parameters to read appropriate data from test data repository. Test Data Repository: A place to store , organize and manipulate test data. It is usually made up of any one or combination of flat files, excel workbooks, relational databases. This is also where results of the automated test execution will be stored. The idea is to have a centralized location for controlling test data and for reporting. Driver: It is a batch test which calls other automated tests. It will have error handling/recovery capability where if one test in the batch fails to run due to any error, that test will be ignored and execution control will move to next test in the batch. Components of the Functional Decomposition Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 23. 23 Appendix A - Structure of Business Functions An Automation Framework That Really Works - Basivi Reddy Junna <Function Name>(Arg1,Arg2,...) ‘Verify if the test failed in previous function If STATUS <>"PASS" Then Exit Function End If 'Validate parameters passed to the function 'for example you verify if they are null. or in some cases verify if they are numbers or not etc 'If IsNull(Arg1) or IsNull(Arg2)... Then 'Perform appropriate action based on whether any argument is mandatory for the function to continue 'Report about missing or incorrect argument and Exit Action 'End If Validate Start Status 'Describe in what state the application should be (screen name and short description) when this function starts 'Create Expected Result (ER) strings for all conditions this function seeks to verify 'Expected Result1=“…" 'Expected Result2=“…“ write code to verify each Expected Results ‘Wherever necessary implement synchronization, parameterization, Error handling, Insert output values and checkpoints 'Capture Actual Result for each Expected Result ; Compare Expected and Actual Result 'Report Pass if they match or Fail if they don't along with expected and actual results ‘While performing above steps recover from errors if any and also call Common functions as needed <End Function>
- 24. 24 Example Setup An Automation Framework That Really Works - Basivi Reddy Junna Setup file - Setup IDs sheet Setup file - Setup Details sheet
- 25. 25 Example Setup An Automation Framework That Really Works - Basivi Reddy Junna Setup file – Run Setup IDs sheet Setup file – Tests Sheet Setup file – Run Tests Sheet
- 26. 26 Sample functions for a fictitious E-Commerce Application An Automation Framework That Really Works - Basivi Reddy Junna
- 27. 27 Functional Decomposition Framework Sample Test and Results An Automation Framework That Really Works - Basivi Reddy Junna
- 28. 28 Workflow of the Functional Decomposition Framework An Automation Framework That Really Works - Basivi Reddy Junna
- 29. 29 Functional Decomposition Framework - Driver
- 30. 30 Functional Decomposition Framework – Run Test
- 31. 31 Workflow of the Functional Decomposition Framework – Test Data Repository Considerations An Automation Framework That Really Works - Basivi Reddy Junna Systems/ Applications With UI No UI (Headless)
- 32. 32 Workflow of the Functional Decomposition Framework – Test Data Repository Considerations An Automation Framework That Really Works - Basivi Reddy Junna UI Based Applications Free form text type Text box Combo box Text area Single Select Type Single select list view Single select tree view Radio button group Dropdown list Multi select type Multi select list view Multi select tree view Checkbox
- 33. 33 Workflow of the Functional Decomposition Framework – Test Data Repository Considerations An Automation Framework That Really Works - Basivi Reddy Junna Non UI Based /Faceless Applications Single values Set of values
- 34. 34 Workflow of the Functional Decomposition Framework – Test Data Repository Design Recommendations An Automation Framework That Really Works - Basivi Reddy Junna Data Tables Static Data Tables Contain static data . Examples: list of states, product categories like furniture, electronics etc 1. For each set of static data create a separate data table and assign unique Id for each row 2. Use these IDs delimited by ; inside dynamic tables if the function needs to handle multi select type fields or a set of values as input 3. Use the static data values directly inside the dynamic tables if the need is to handle single select type fields or a single value as input Dynamic Data Tables Made up of: 1. Free form text, numbers and dates 2. Different permutations and combinations of data from static tables
- 35. 35 Test Data Repository Design - Sample An Automation Framework That Really Works - Basivi Reddy Junna Table to store values captured from Application
- 36. 36 Pros: • More flexible than other frameworks • Better maneuverability means test flows involving complex twists and turns can be created easily • Better control over test data consumed in your tests • The building blocks (functions) are not too granular, not too high level and present fewer points maintenance • One of the best frameworks suitable for automation in agile environments Cons: • Becomes complex when the test gets big - as the test size increases, so will the number of function calls and failure points, debug points • When a test becomes really big, test flow is not as clearly evident as in the of modular framework Functional Decomposition Framework An Automation Framework That Really Works Basivi Reddy Junna
- 37. 37 Hybrid Framework(s) An Automation Framework That Really Works - Basivi Reddy Junna Keyword + Functional Decomposition + Modular Framework Keyword + Functional Decomposition Framework Functional Decomposition + Modular Framework Keyword + Modular Framework
- 38. Do not copy or distribute. Proprietary and Confidential to Scalar USA Inc. 38 Q&A ??? An Automation Framework That Really Works - Basivi Reddy Junna
- 39. Do not copy or distribute. Proprietary and Confidential to Scalar USA Inc. 39 Thank you!!! [email protected] An Automation Framework That Really Works - Basivi Reddy Junna