Advanced Test Design Methods

• What is a Perfect world?
• Designer in a Perfect world
• Testing in a Perfect world
• Design down to Earth
• Business is King
• Simplicity &Complexity
• The Data factor
• Risk based Design
• Next Generation
Advanced Design Methods

What is a Perfect World?
Business People never change their mind
Methodologies always assume the world is perfect:
Documents are always clear and complete
Executing TCs people read every word
Documents are always updated
The solution addresses all needs
Communications flow is immediate
We review to present more then to approve
Design can be automatically created from Requirement documents

Designer in a Perfect world
The Tester is the first user to experience the system
No Amount of Requirements definitions and pre-
planned Automatic tests could ever replace the
inputs coming from a User who looks at a screen
While the defects are considered Low Severity they are the ones which Prove SI value
 How are the Icons, Logos, Fonts, Details representation shown on a screen?
 Is information presented to the customer understandable when trying to perform
an action or overcome a mistake?
 How high is the Density of information presented to the Eye at any given moment?
Even in a Perfect world:
Smart Manual Design can guide a Tester to investigate efficiently the solution

• Where is the Designer’s
Value?
• Aspects of a Smart Design
• Design from Start to Finish
• Simplicity & Complexity
• The Data factor
• Next Generation

Where is the Designer’s value?
The world is far from perfect!!! And should never strive to be!!!
Overall goal of the Designer:
Plan tests which will produce the confidence needed to take decisions, and can be
executed within given timeframe.
Secondary Crucial Targets:
• Smart and Efficient
• Be Clear and Realistic
• Guide the Executer to think
• Presentable Design
Creative

Aspects of a Smart Design
• All effort invested in planning/design gives value to the execution.
• Read documents while extracting points for design
• Give the executor a single design document
• Collect all non execution information to the strategy
• Edit all Reused materials to fit your project/stream
• Better to keep an activity Empty then detail unclear steps
• Inputs from multiple sources were used early in the design process
• Use Document Center to get background information
• Contact the Developer even before you start reading
• Generate a quick presentable list of your flows
• Share for Review High level design before emotionally
Investing in Detail design
• Dependencies were put only in places where they improve the quality of the test
• Placing two functions in a flow adds a risk of missing one of them
• Limiting to use specific data my delay execution of the test
• Adding an external system validation forces integrated env
• Running batch activities requires alignment

• Tests added only in places they increase the coverage of the scope
• End the flow when it’s value ends
• Avoid copy/past of the same validations
• Every flow has a clear business purpose
• Manage a data coverage table to be sure all is used efficiently
• Tests added in to a calendar have both Technical & Business reasoning for grouping
• Start Design by Defining the scope of each calendar
• Never fear from too small calendars
• Split Calendars based on Business scope
• Split Calendars based on large Batch dependency
• Look for Common Functions based on Business Process
• Risk calculated by the actions the user will do and not what system allows
• Add tests based on Risk to the Business
• Prioritize based on Customer inputs and Usability
• Concentrate on realistic Business Use cases
• Always keep in mind the Customer’s Master target

• Retesting full flow must be simple and feasible within reasonable timeline
• Flows with multiple targets are harder to retest
• Flows with time dependency should be very focused
• Flows length must fit the timetable provided
• Consider possible WA options during the design
• The execution of the Designed tests feet the timeline and resources of the project
• Prioritize so if limitations arise change would be quick
• Estimate at least 15% of execution as added Ad-Hoc
• Design the amount of tests to leave contingency
• Order the tests based on Priority
• Make sure Value output is reached at all points of execution
• The Designed TCs please the Customer
• Start Design after understanding the high level customer’s goal
• Search for inputs the customer gave earlier on
• During Review collect all customer inputs
• When revising include all Customer inputs even at expense of
other tests you feel are important
Quality
Time
Resources
Scope

Design from Start to Finish
1. Gather Inputs:
DC Docs Talk to Dev Previous AL Customer perception
2. Analyze Requirements to Extract Use Cases:
Describe the User experience
3.1. Breakdown the
Detailed Activities:
3.2. Share your
view:
Review and UpdateBuild Activity Library
4. Design Full flows:
Describe highlights Map data Requirement Coverage
5. Detailed Design:
Build Calendar in ATS Export to QC Generate Report in ATS

• Business Process at all
Design levels
• Customer Centered – what
makes your Account Special
• The Data factor
• Next Generation

Business Process at all Design Levels
Business Use Cases
Example:
A Subscriber Migrate a SOC with Multiple subscribers, Add Multi SIM, Buy Data Pass, Usage query on customer with
multiple subscribers, free and counted traffic
Concept:
A high, high level description of the flow, focused completely on the functions tested , addressing information in
business language only.
Purpose:
Allows The designer to extract quick valuable Testing output which can be reviewed by Business Units to provide their
perception of the testing scope. Gaining feedback at an early stage allows the designer to incorporate the remarks and the spirit
of the Business outlook as the designer still did not have time to get attached to his own concepts of what’s important to test.
Method:
 Start from the Business Requirements, gather the functions which need to be tested in to a diagram detailing the flow within
all known business processes (COP, CHQ, PLC, MRL, CCS, BDA, CP, CDNP, ATC, MPR).
 Break down the diagram to it’s different flows and describe use cases for every Top Risk permutation of the flow.
 Each Requirement must be reflected in at least one Use Case, Requirements with conditions or parameters will require
multiple use cases to cover them.
 Information form multiple requirements can be gathered to one Use case but make sure to do so only if it gives value to your
test
 Every Business Use case must detail 3 sections:
 Data origin – New, Converted/Migrated or just Select – if selection has no added value, create new to be
sure the tester will reach the important functions
 BP functions – combine between BPs if there is specific value to it, if not remain within the BP
 End point validation – always use the user’s real method of validating (online query or a next action)

Business Process Layer for Activity Library
Example:
Concept:
An Added layer of Activities in the AL which joins activities from different applications to describe the full business process. It
doesn’t contain end steps, but links to the original activities in their Application tree. The BPAL layer keeps the order of activities
using sequences
Purpose:
Provide designers with a clear view of each business process without a need to lookup in each application. Allow reuse of the same
activity for different business processes. Quick modification of BP level customizations when the base activities remain the same
Method:
Once the Application level Activities are created open under the main AL directory a Directory named ‘_BP’.
Under it place directories for each BP. Under each BP create sequenced directories for each section in the BP.
Create Sequenced TCs for each Activity in each section. If there is more then one option for an activity, place the
same Sequence. Advance the sequence based on the steps in the process. Optional steps must be marked.
Use the ‘Call to Test’ to link to the original activity – so each activity in the BPAL contains just one link
CRM
Customer Activities
Billing Arrangement Activities
Subscriber Activities
Customer Creation
Individual Customer Creation
Business Customer Creation
Corporate Customer Creation
Customer Maintenance
Description Expected Results
1 Open CRM Interaction
overview shows
2 Click Menu 
Create  create
Customer
Create Customer
screen opens
3 Populate first
name
Field accepts
minimal 2 chars
… ……………………. ……………………..
OMS
_Provide Order
Change Order
Suspend Order
001_Start Order
002_SOAP – Select Order Action & Product
003_NIA – Negotiate Installation Address
004_NPC – Negotiate Product Configuration
005_NCD – Negotiate charge Distribution
………
COP
01 Customer
011_Individual Customer Creation
011_Business Customer Creation
011_Corporate Customer Creation
02 Financial Account & Billing Arrangement
021_Create FA and BA - CC
Description
1 Go to Individual
Customer Creation
03 Order
031_Start Order
032_SOAP – Select Order Action & Product
033_NIA – Negotiate Installation Address
034_NPC – Negotiate Product Configuration
035_NCD – Negotiate charge Distribution
………
04 DB Verifications
ATC
CHQ
BDA

Business Process Oriented Testing flows
Example:
Concept:
A high level description of a flow representation of the Business Use cases, done either in word or excel and is later also used as
base for test results documentation. Provides the reader complete understanding on ‘What it is we Test’ leaving out only details
related to ‘How we test’.
Purpose:
Presents to Reviewers and the Executer all the details needed to understand our testing plan and the method in which we cover
our scope.
Method:
Breakdown the Use Cases in to detailed flows, each use case must be represented by one scenario. From the
Development HLDs extract details related to the exact sub steps of each action and the many parameters which
must be covered to insure a function works. Detail in the bullets of the flows the following types of information:
 The System being used
 The function executed in the test
 The values which need to be selected to perform the test – focus on values which are needed for generation of
the intended flow.
 A validation which can be done as a direct result of the action performed (no added action needed) – validations
which require an added action should be detailed as a separate bullet

• Advantages of each
approach
• Balance Point
• The Data factor
• Next Generation

Bill
Advantages of each Approach
The Business Process Oriented Approach:
Examples:
When to Use:
Strong Points:
Order Use Change Use Bill Complain UseChange
 Best use when expecting unstable behavior and high rate of Critical issues.
 Types of Activities for which this approach works well for are:
 Green Field project
 Conversion Project
 New Business Line on existing System
 Large Customization with dramatic CRs
 Defect Retesting portion of a Maintenance Delivery
 Allows to see and reflect to the customer clear results for each function – to facilitate difficult
decision making.
 Enables Defect retesting based on Full flow – to insure recreation of all conditions contributing to
the previously defective flow
 Permits running all flows in parallel giving full system indication of quality even if some functions
have critical issues
 Allows Focusing and re-running Specific Business Processes at higher rate then others
/ / /

Re-Bill
Advantages of each Approach
The Full length, all in one Approach:
Example:
When to Use:
Strong Points:
Order Use Complain Change Use Bill Complain UseChange
 Best use when expecting flows to pass without generating damaged data – meaning expected
errors are on screen or flow levels, meaning each individual action will work but the flow will
not end with the expected result (for example event is rated but with wrong rate)
 Types of Activities for which this approach works well for are:
 Regression for stable System
 Large number of Small GUI CRs touching the same Business Process
 Allows to see a real, full, user experience of the system
 Makes full use of each Resource, reusing the data created by one action as base for the next thus
minimizing the number of un-effective test cases for data creation.
 Insures Higher confidence defects related to complex behavior between seemly unrelated function
are identified during testing
 Provides the ultimate validation each action works by proving the data can be used for added
activities without error

Balance Point
Simplicity Complexity
Accurate results User Experience
 Make sure flows can be executed within the time given for Execution of a test round
 Make sure the scenario name reflects the tests in the scenario – so managers who read the
report will be able to understand what works and doesn’t work
 Make sure all parameters related to a function are covered, including data types which
require a previous action to be done to generate the data
 When there are two Options to generate the same end state of a test, choose the shorter
option.
 Combining unrelated tests which don’t give real value to each other must be avoided
 Join to one Bullet actions and validations when the validation is a direct result of the action
 Hint to it when expecting the user to perform look and feel general validations.
 A validation should be added only if it’s giving a real new input – avoid Copy/Paste

• Present Coverage
• Allow Flexibility for Change
• New & Existing data conflict
• The Data factor
• Next Generation

Present Coverage
Implementation & Applicative data Coverage can be presented in two levels:
 Full Coverage – for parameters which require a small exact specific list
(Examples: Rate/Allowance PITs, Customer Type)
 Cover by Groups – for parameters with large range of data, grouping by Rules or by
Type
(Examples: Service filters grouped by Event Type, Customer Address Grouped by Country)
Coverage Matrix
Built to show in one view the data types covered during testing. Focused on the
Parameters being tested.
Constructed from the Parameters encountered during the flow of each Business
Process.
Recommended Structure:
*Update Parameter names and values and add the covering Use cases during the design

Allow Flexibility for Change
Real Data is always unstable:
• Customer Security Policies often prevent from publishing Real Implementation early
• Production Implementation is a living, changing element for any Company striving to be #1
• Last Minute Regulations can cause unexpected change in Implementation
• Delay in Timeline introduces new, unexpected Implementation and Applicative data in to play
• Applicative Data is forever changing both in values and in Types
• The Go live date determines different types of data reconciliation during conversion projects.
Methods to ease coping with change:
Place Final values only in places the specific value generates the flow
Use Indexes from Gathering Tables instead of final values when it is not absolutely certain
the list if final
After reaching full coverage of a Parameter allow Random selection during execution
At any Delay in timeline approach BPT team to find if any updates were made in the
implementation gathering
Refresh Data from Production at least once during any version which exceeds 3 months
For Migration flows Prepare Extra Designed flow for scenarios which depends on Go live
date, keep the designed scenario out of the delivered document, as safety for delay option

The New & Existing Data Conflict
Why Should we Create our own Data:
 Designer is in Full Control of the Data used
 No Dependency on Stable Data from Production/Previous
 Synchronized in all Systems
 Allows Use of Test Cards for physical device tests Fresh
Why Should we Use Production Data:
 Test is Meant to Test Production data Conversion
 Quality Defects Benefit from Real Full Customer situations
 Quality Defects from local language data
 Quality Defects from long existing history data
 Shortcut for long process data creation
Real
How to Chose:
 Using Existing Data must have a reason – if there is no reason go with New
 If a testing target was achieved by a different scenario using existing, use new
 When testing Conversion New data can still be used to test converted functions
 Plan the scenarios which require existing data first, and then move to the
functions which can be tested on newly created data

• All Design is Risk Based
• Prioritization Factors
• The Data factor
• Next Generation

All Design is Risk based
It is Impossible to do every possible test
 The Customer Lifecycle is a never ending Story
 A single order can go to Loops of Back , Next and Amend
 A large group of small parameters generate a huge testing matrix
 Execution Time is limited and LD is not always supported
Choice introduces Risk
 Critical Tests or entire functions might be missing from the design
 Focus could be placed on un important elements
 Impossible situations could be designed when trying to minimize
 Tests could go far from the customer’s main use cases
 Covering matrixes might not leave time for creativity

Prioritization Factors
1. Main Business Target related Use Cases
2. Creative Use Cases combining multiple related functions
3. Use Cases targeting race conditions of data flow between systems
4. Use Cases targeting race conditions of data flow dependent on time
5. Use Cases aimed for Edge particular populations
Follow the Business Use Cases as Top Priority:
What if we can’t cover all values of a Parameter?
1st Priority: Most Commonly Used in Production
2ed Priority: Most likely to assist in introducing Complexity to the flow
3rd Priority: Reproduces a known former Defect
Surely Avoid:
• Hasn’t got even 1 existing case in Production
• Has been declared by Business as out of scope of the Function tested
How to make sure our focus is ‘Acceptance Test’?
1. When checking coverage matrix avoid adding too many tests on Unit test level
validations just to complete the coverage
2. When designing a Test always keep the User in Mind, ask yourself if the
created test is something that can happen to a User

• The Data factor
• Next Generation

Are you Tired of Testing?
Tired of writing design no one ever Reads?
Frustrated with Unclear descriptions from
your designers?
Don’t want to spend so much time on Reading
When you should be executing?

Come See the Revolution
The Revolution: Visual tests, comparing ‘apples’ to ‘apples’
Passed
Passed
Failed
No Run

Visual Testing Method
Design Process
In the Tool:
Gathering Generic
Activity Library
Of Mockups
Out of the Tool:
Designer Maps
the Scope & Selects
the Use Cases
In the Tool:
Designer creates
for each use case
a mockup flow in
the tool with tests
Defined at each
mockup
* Phase 2 will provide a solution within
the tool for Mapping of the scope and
mathematically selecting Use cases
In the Tool:
Automatically
Export TCs to
Quality Center
Keeping a link
to the tests in
the Tool

Visual Testing Method
Execution
The Executer places the Tool’s View on one screen and the System on the other
The tool describes visually the tests to perform and allows to Pass or Fail a test within the tool
Passed
Failed
No Run
No Run
*
Passed
Failed
No Run
No Run
No Run
* Fully Integrated with QC

Phase 2 – Shooting over the Moon
Business Process Integration within the Tool
Enhancement 1:
Generic Sub Flows as Elements of BP Activity library
Enhancement 2:
Diagram flow description of the full BP tree
Enhancement 3:
Automatic Pairwise Selection of Optimized Use Cases

Advanced Test Design Methods

Recommended

More Related Content

What's hot (20)

Similar to Advanced Test Design Methods (20)

Advanced Test Design Methods