Alex syvorotka - QA: Customer Oriented Testing Approaches in theory and practice
Download as PPTX, PDF1 like900 views
This presentation is intended to share the experience around customer oriented testing gained in one of Lohika projects. It will describe how important such approach is, what are the outcomes and which tools, processes and technics can be used.
Ad
More Related Content
What's hot (20)
Similar to Alex syvorotka - QA: Customer Oriented Testing Approaches in theory and practice (20)
Ad
More from Ciklum Ukraine (20)
Ad
Recently uploaded (20)
Alex syvorotka - QA: Customer Oriented Testing Approaches in theory and practice
- 1. Customer Oriented Testing Approaches in theory and practiceby Alex Syvorotkaformer SQA @ Lohika2011Lohika Confidential
- 2. ContentsPresentation Description and Short words introHP UCMDB Quick OverviewCOST aimsMethodologyProgram stagesOn-Site visitProgram requirementsCOST testingMet goals08/20/11Lohika Confidential2
- 3. Presentation DescriptionThere are many approaches in Quality Assurance are being used in enterprise software development which include many different testing types. Starting from classic functional testing and going beyond with different techniques of non functional testing.This presentation is intended to share the experience around customer oriented testing, aka COST, gained in one of Lohika projects – HP UCMDB.It will describe how important such approach is, what are the outcomes and which tools, processes and techniques can be used.08/20/11Lohika Confidential3
- 4. Short words introEvery workday QA tries to provide testing coverage of software components by test cases as full as it possible. In order to make coverage really high HP has agreement with premier customers to make EA cycles during on-site visits. And along with other benefits of such visits - to retrieve dumps of their software in production. Obviously owing to security reasons customer hides vulnerable details such as credentials, business critical pieces of information, etc.QA analyses DB dumps to clarify in what way (HOW?) particularly a certain customer uses HP UCMDB (BSM). QA creates tables\maps of: DB capacity, used configurations, amounts. This allows to use replicated customer’s environments for testing and also to understand which of the configurations is the most effective for a specific area of testing coverage.Cross-team testing – a method applicable to mass testing of software product components. The main idea is to make a joint effort of all teams to cover their areas of responsibility. An experienced QA guy is being challenged as a cycle owner. He prepares cycle time periods, focus points, servers, presentation. And, if necessary, sheds a light onto possible technical issues and “bottlenecks”. The owner assigns details of the cycle and tasks to participants and sets timelines. As a result of cross-cycle an aggregated report is generated and sent to management of the particular area.08/20/11Lohika Confidential4
- 5. HP UCMDB Quick overviewThe complexity of today’s IT infrastructure places enterprises at great risk of prolonged and frequent IT outages as a result of change. For example, market research has found that upwards of 80 percent of IT outages are caused by human error that stems from both planned and unplanned IT changes, as well as shortfalls in testing and process inadequacies. Yet simply avoiding change is not possible. Things break. Software needs patches. Servers will always need memory upgrades.Agility in IT means responding to business needs when a company builds, buys, divests or merges lines of business. It is defined by the ability to quickly deploy and manage new services for example, a Web application that the business launches in order to pursue an emerging market opportunity and stay ahead of the competition. IT’s role as a strategic business partner stems from its ability to make changes quickly – and less risk. If change is mandatory, then IT organizations need a means to control change and manage the impact of change when it does occur. This is the value proposition of a CMDB. 02/20/09Lohika Confidential5Painful process called “Change”
- 6. HP UCMDB Quick overviewA Configuration Management DB visually models information about IT infrastructure components in order to understand the interdependencies among these components in the context of IT service. If IT understands the relationships among its IT components – if it has the ability to map components to applications to the business – then IT can control change and manage impact. IT needs to understand the impact of change prior to implementing changes in their production environment. For example, IT needs to understand impact the installation of a new O/S software patch to a single box in a server farm will have on a business service – such as online trading – and can take proactive steps to mitigate the risk of an outage. Similarly, should outages arise from unplanned changes or for any other reason, such as fault failures, the relationships the CMDB maps allows IT operations to quickly find the root cause of IT outages or degradations – often before performance, availability or end-user satisfaction are adversely affected.02/20/09Lohika Confidential6CMDB’s Purpose
- 7. HP UCMDB Quick overview02/20/09Lohika Confidential7CMDB and discovery relationships
- 8. HP UCMDB Quick overview02/20/09Lohika Confidential8
- 9. HP UCMDB Quick overview02/20/09Lohika Confidential9
- 10. HP UCMDB Quick overview02/20/09Lohika Confidential10
- 12. COST aims08/20/11Lohika Confidential12Test products from the customer’s perspective, using real business scenarios executed on authentic dataProvide version status as reflected on customersBring customer’s use cases expertise in house to improve prior release testing
- 13. COST MethodologyUnderstand customers usage, needs and painLearn customers methodology and implementationSimulate customers deployments using real dataExecute customer oriented scenariosFocus R&D on the real usage of our products08/20/11Lohika Confidential13
- 14. COST Program Stages08/20/11Lohika Confidential14Visit customer site
- 16. Understand the organization testing process
- 17. Receive feedback/enhancement requests directly from product users
- 18. Retrieve Product DumpCOST Program Stages Cont.108/20/11Lohika Confidential15Replicate customer environment using real data
- 19. Document business processes and usage scenarios
- 20. Validate informationCOST Program Stages Cont.208/20/11Lohika Confidential16 Upgrade environment and data
- 21. Run user scenarios
- 22. Validate new featuresCOST Program Stages Cont.308/20/11Lohika Confidential17Install new version in QA Laboratory
- 23. Perform upgrade over customer’s dumps
- 24. E2E cross teams testing
- 25. Massive bug hunt on customer’s dumpsCOST Program Stages Cont.408/20/11Lohika Confidential18Install new version on site
- 26. Perform upgrade
- 27. Post upgrade tests with usersOn Site Testing ActivitiesSelecting areas for ‘on site testing’Critical areas that a large percent of customers are exposed to.Areas containing high number of fixes or where refactoring has been done (risky areas).Areas that suffer from ‘bad reputation’ (are more prone to defects) By demand – from QA or Dev02/20/09Lohika Confidential19
- 28. On Site Testing Activities Cont.1Planning the testing activityAnalyze the feature with relevant QA FL and Dev FL; understand current problems and risk factor related to them.Get buy in of Dev (CORD, feature team) and set criteria for build that will be used for this activity (including fixing QA holders)Determine environment coverage needed for this feature (DB, OS and etc.)Identify actions that can be done on customer site and verify that these actions cover the risk areas.Create specific use cases to be tested at each customerLearn the feature thoroughly including troubleshooting knowledgeSelect between remote activity and on site activitySet available support level for customers participating in this activity.02/20/09Lohika Confidential20
- 29. On Site Testing Activities Cont.2Selecting customers – requirements and profileCustomer characteristics: environment parameters (if relevant to tested feature), projects structure, special configuration, challenges and problems and etc.Customer is willing to cooperate and accepts relevant requirementsMatch between feature and usage patterns at customer site.Location (when relevant)02/20/09Lohika Confidential21
- 30. On Site Testing Activities Cont.3Execution phaseBefore conducting the activity make sure all preparations are done at customer site.Log all defects and problems detected on site, make sure to take all relevant data (logs, configuration).If possible, obtain a copy of production project.02/20/09Lohika Confidential22
- 31. On Site Testing Activities Cont.4Wrap upMaintain support level as agreed with the customer before the activityWrap up communication with customer – provide status of issues detected on his environment.In case of open Support Requests – involve CSO and CORD.02/20/09Lohika Confidential23
- 32. COST Testing ProcessRequirements definition (obtained during customer’s visit)Installation and configuration UpgradeCritical business scenarios per projectTesting scenarios and features validationBusiness flows by user roleE2E testing08/20/11Lohika Confidential24
- 33. Met goalsGet overall understanding of HP UCMDB implementation at the customer.Understand challenges faced by the customer.Define specific testing requirements for upgrade and critical areas in HP UCMDB.Obtain project specific business processes to be translated into testing scenarios.Create somewhat like Customers Knowledge Base08/20/11Lohika Confidential25
- 34. Q&A?WELCOME TO LOHIKA!Lohika Confidential2608/20/11