QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных сторон. Реальное применение

QA Metrics in Agile
Look at the Quality from different angels. Real appliance.
By Volodymyr Prymakov
www.qaexperts.pro

About Me
Volodymyr Prymakov, Ukraine
Senior QA Manager at Ciklum, TCoE
Head of Performance QA Unit at TCoE
Interim Head of TCoE at Thomas Cook
•14 years in QA
• >45 projects experience
• Certified: ISTQB Advanced Test Analyst &
Manager, ICAgile CP, SAFE
/in/vladimirprimakov/
www.qaexperts.pro

What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro

What is quality of Process
Quality Process brings much Business Value with keeping project or program engine safe and efficient
1. Deliver new functionality often
2. Deliver more functionality
3. Delivery quality software
4. Delivery what is planed
5. Deliver with minimal risks
6. Business trusts dev teams, processes, and produced results
www.qaexperts.pro
1. Project vision, goals, roadmap, and other plans are clear to everybody
2. Process is transparent and strait-forward
3. Product quality is always transparent
4. Technical dept is being eliminated
5. Efficient Communication and Interdependency solving
6. Shared decisions making and responsibility
7. Lessons learned and continuous improvement are in place
8. High team sprit and satisfaction

Filters/Bases
for
QA Metrics
Measuring Product
Quality
www.qaexperts.pro

Definition of Done (DoD) defines
Which quality Product should
have?
Which Way and How
thoroughly a product or
functionality should be
tested?
Min Sufficient Max Allowed
www.qaexperts.pro

Definition of Done for Stories, e.g.
• Coded and integrated to main branch,
• Unit tests are written and passing,
• API component tests are written and passing,
• Code reviewed and conforms to coding standards
Passed Dev
Testing:
• Code deployed to QA environment. Automatic API-integration and
EndToEnd smoke testing tests are passing,
• Test Design is completed,
• Positive and negative Acceptance Tests are (automated and)
passed on platforms of priority 1. Exploratory tests are passed on
platform of priority 2.
• Selective Regression is carried out by a necessity and passed.
Passed QA testing:
No (P1/P2)
bugs
outstanding
for STORY
and neighbor
functionality
Story meets INVEST criteria
Story is Signed-Off by Product Owner
www.qaexperts.pro

Definition of Done for Sprint, e.g.
• Stories that meet DoD are considered DONE, the rest goes to
next sprint.
• It should be avoided to merge Risky or buggy code to main
branch at the end of sprint!
Stories meets DoD
• Regression testing is passed with P1 test cases on at least 1
top priority platform/device
• Smoke or exploratory testing is passed on platforms/devices
of priority 1 and 2
Cover Regression
Test
• Main Integration tests for live 3d-party services are passed on
STAGE or other environments3d party Integration
Test
No (P1/P2)
bugs
outstanding
for product
and neighbor
products.
Sprint is Signed-Off by Chief Product Owner
www.qaexperts.pro

Definition of Done for RELEASE, e.g.
• All Stories, which were merged to Main branch, meets DoD for stories.
• Risky or buggy code is not merged to main branch or is toggled off or is
reverted!
Stories meets DoD
• Regression testing is passed for all priority 1 platforms/devices.
• Exploratory testing is passed for priority 2 platforms/devices.
• Smoke testing is passed for priority 3 platforms.
Full Regression Test
• All Integration tests for live 3d-party services are passed on STAGE or
other environments
• There is no other 3rd party release ongoing that may impact release
negatively
3d party Integration Test
and Releases
• UAT is passed on STAGE environment
UAT
• Performance Testing successfully passed
• Security Testing successfully passed
• and other types of testing and preparations to release
Other Types of Testing
No (P1/P2)
bugs
outstanding
for product
and neighbor
products
Release GO Aproval from by Chief Product Owner, QA team, Release Manager and
respective Business representatives
All other unresolved
bugs are tracked,
reviewed, and planned
when to be fixed
www.qaexperts.pro

Definition of Done
dictates you
which Activities and QA metrics
to perform and monitor!
www.qaexperts.pro

Types of Metrics
Quantative - Information that is directly measured as a
number
Qualitative - Information that results from human
observation and analysis
www.qaexperts.pro

Types of Metrics
1. Absolute metrics
2. Relative metrics
3. Derivative metrics
Purpose of Metrics.
1. Goal Metrics: To measure results/goals
2. Actionable metrics: to make decision
3. Predictive metrics: to predict future
www.qaexperts.pro

Stages at which QA and other metrics are needed
Implementation period
• Daily Monitoring of
Quality, product
implementation, and
test progress
Key sprint and release
points
• QA Report Before
sprint Demo
• QA Reports During
stabilization /
regression / UAT
period
• QA report Before
Release
• QA Reminder report
when product has
critical quality state
or low product
implementation or
testing progress
After Release
• Release quality and
process analysis
• Quality and process
Trends and analysis
• Team efficiency
analysis
www.qaexperts.pro

QA Metrics
for
Daily
Monitoring
www.qaexperts.pro

Goals of Daily Quality Monitoring
1. What is the current quality of the
product?
2. How much work is left?
3. Do we have enough time to finish and
stabilize the product?
www.qaexperts.pro

!!!Important:
Breakdown Defect Metrics
Always breakdown your metrics to the following groups:
All Bugs! – shows general product quality state
Critical and Blocker bugs – Shows whether we
meet Readiness criteria!
Lower Priority bugs– Shows the rest of bugs,
general amount of which can negatively influence on the
end-user’s impression about the product.
www.qaexperts.pro

Defects reported by Date
Critical and Blocker bugs Major and lower priority bugs
Jira “Recently Created Chart” Gadget
Allows you track new bugs and immediately react to them!
www.qaexperts.pro

Open Defects reported Today
Jira “Filter Results” Gadget
Allows you to quickly review new bugs
www.qaexperts.pro

Open Defects By Component/Platform
Major and lower priority bugs
Jira “Pie Chart” Gadget
Critical and Blocker bugs
Shows which
component/platform
is not ready for release
or requires bug fixes
www.qaexperts.pro

Defects By Component/Platform for Current
Affected Version (any Status)
Major and lower priority bugs
Jira “Pie Chart” Gadget
Critical and Blocker bugs
Shows which
component/platform
require more thorough
testing,
e.g. (Selective)
Regression testing
www.qaexperts.pro

Critical Open Bugs by Status
Jira “Issue Statistics” Gadget
Allows you to quickly get Status of Open Critical Bugs and understand whether we have enough time to fix them
www.qaexperts.pro

Open Bugs by Priority
Shows general quality of the product
www.qaexperts.pro

!!!NOT-assigned Open Critical Defects
To Fix Version
To Team
(for big teams only)
To Sprint To Developer
Shows probably lost or forgotten Critical open bugs, which should be processed or escalated immediately!
www.qaexperts.pro

www.qaexperts.pro
Workload:
Assigned Open Tasks and Bugs for Developers/QAs
By teams By Developers
For current Fix Version or sprint
By QAs
Stories
By teams By Developers By QAs
Bugs
To assure that there is no bottlenecks and overloads. To Remind about
work assigned!
To spread load equally between teams, Developers and QAs!

Release scope status by developers/QAs/teams
If by the end of sprint or before stabilization period at release a lot of stories are in open status,
it can bring risk of late merge status, not enough time for testing, not ability to finish functionality etc.
Action: Try to spread load equally by teams, QAs and Dev Specialists. Remind about work Not Done.
Postpone certain functionality to next Release!
Stories Bugs
Jira “Two Dimensional Filter Statistics” Gadget
www.qaexperts.pro

Stories/Bug Status By Fix Version
created >= -365d
Stories Bugs
created >= -180d
Allows to understand ability to finish/fix all tasks or bugs in time for a certain sprint/release
Jira “Two Dimensional Filter Statistics” Gadget
www.qaexperts.pro

Current Fix Version Progress
Shows general release/sprint ProgressJira “Release Report”
www.qaexperts.pro

Team Burndown Chart
www.qaexperts.pro

Kanban board for All Open Critical and Blocker Bugs
(For big teams)
Allows to
accumulate
and track all
open Critical
and Blocker
bugs in 1 place
www.qaexperts.pro

Regression testing
Recommended Statuses:
• Failed with Critical and Blocker bugs
• Failed with Major and lower priority bugs
• Failed by automated tests
• Blocked
• Not tested
• In Progress
• Passed
www.qaexperts.pro

Regression Testing Coverage and Results
For Release/Milestone
– General Results
By Cycle/Stages
- Every cycle should has certain purpose, e.g:
1. Initial Feedback about quality,
2. Exploratory testing,
3. Confirmation testing
4. 3rd Party services testing.
www.qaexperts.pro

Regression Testing Coverage and Results
By Product Component
• by platform/device
• by test priority:
• Priority 1
• Priority 2
• Priority 3
By Type of Test:
• Smoke test
• Regression test,
• UAT test
www.qaexperts.pro

QA Metrics
For Regular
Quality Data
Reports
(Snapshots)
www.qaexperts.pro

Sprint Report
1. Story Delivery rate % = Actual Stories Completed vs. Committed Stories, e.g.: 90%
2. Amount of Stories met DoD, e.g.: 25
3. Amount of Stories Moved to a new sprint, e.g.: 4
Because of 1 not implemented, 1 Reopen because of Critical Bugs, 2 Not Tested
Specifies the reasons, risks and recommendations.
4. Regression testing coverage and results: Graph
5. Cross platform testing coverage and results: Graph
6. 3d-party integration testing results: Graph or written explanation.
7. Testing and testing activities NOT done – describe reasons, connected risks, and recommendations.
8. Open Bugs:
All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link to jira)
Open Major bugs opened during the sprint, e.g: 40 Major bugs open during sprint. (link to jira)
9. Point to the buggiest product component and platform: Graph and explanation
10. General Conclusions
www.qaexperts.pro

Regression testing reports, daily or by cycles
Reports about Regression testing, cross platform testing, 3d-party integration testing, 3d party releases testing, etc.
1. Name of the Regression testing stage, e.g.: Regression Cycle 1 (Preliminary feedback)
2. Coverage and results: Graph
3. Open Bugs
New (created today) and All Open Critical and Blocker bugs:
New 4 Critical&Blocker Bugs, (All 5 Critical&Blocker Bugs) (list or links to jira)
New (Created today) and All Open Major bugs opened during the release:
New 10 Major bugs, (All Major bugs created for current release) (links to jira)
4. Specify Time needed to finish regression testing and/or necessity in additional regression and other testing.
5. Point to the buggiest product component, platform, or functionality if needed.
6. Recommendations
www.qaexperts.pro

Release Readiness testing Report (by necessity)
1. Specify functional testing accomplished and its status:
Regression and cross-platform testing,
3dparty integration testing,
UAT
e.g.: Written explanations and Links to Test management system,
or Graphs of testing coverage and results.
2. Testing and testing activities NOT done – describe reasons, connected risks and recommendations.
3. Specify Non-functional testing status, like performance and security testing.
Whether the product is ready or not ready from non-functional point of view.
Describe connected risks and recommendations.
4. Open Bugs.
All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link)
Open Major bugs opened during the release and All Major bugs, e.g: 40 Major bugs open during release.,
70 Major bugs are Open in general
(list or links to jira)
5. Functionality Not ready for release, etc.: (list or link)+ written explanation
6. General Conclusions!
www.qaexperts.pro

Regression and Release Readiness Report example
www.qaexperts.pro

Reminder Reports (Be necessity)
1. Critical and Blocker Bugs in open state.
2. A Lot of Open Major bugs
3. A lot of Bugs and stories Not Done in the middle or later
stage of sprints: In development, in Test, etc.
4. Availability of not tested stories during stabilization period.
5. Inability to finish testing in time because of certain issues.
Reminder emails or chat messages about:
www.qaexperts.pro

Trend-based
QA Metrics
for
for process
and product
quality
analysis www.qaexperts.pro

Milestones for trends
• AffectedVersion – AffectedVersion field in jira
• Time Span - e.g. period between start and end
date of sprint
Time based conditions in jira queries:
a) createdDate >= '2017-08-30' and
createdDate <='2017-09-12 ‘
b) status changed TO 'READY TO TEST' after
'2017-08-30' before '2017-09-12‘
c) assignee was in (‘user’)
• Sprint – Sprint Field in jira, etc.
www.qaexperts.pro

Found on Environment
Critical
and
Blocker
bugs
Shows leaks of
Critical and
Blocker bugs to
later
development
stages
Pivot Report
www.qaexperts.pro

Found on Environment
All bugs
Shows leaks of
bugs to later
development
stages
Pivot Report
www.qaexperts.pro

Bugs By Priority – Any Status
All Bugs
by
Aff. Version
Shows absolute
release quality
history
– decrease in
number of all
and critical bugs
shows process
improvements
Pivot Report
Note: Should be considered carefully when the team size and product increment increase.
Note: Blank version is bad, and it shows that the Affected Version is not specified for
certain bugs and thus they can be lost www.qaexperts.pro

Bugs By Priority – OPEN (not-closed)
All Bugs
by
Aff. Version
Shows still open
bugs for
releases
– Ideally No
critical and
blocker bugs
should be here
Pivot Report Note: If critical bugs are shown here for previous releases, they might be lost, forgotten or
blocked. www.qaexperts.pro

Bugs by RootCause
Typical Root causes of Bugs:
FrontendOthers
Backend
Frontend
+ Backend
Environme
nt
issues
3d party
components
www.qaexperts.pro

Bugs by RootCause
Critical
and Blocker
bugs
Shows Root cause of all bugs
Pivot Report
All Bugs
Shows Root cause of the most
of important bugs
www.qaexperts.pro

Created Vs Resolved - Cumulative
Critical
and Blocker
Bugs
Created should be less than resolved
Jira “Created vs. Resolved Chart” Gadget
Bug found vs. fixed chart is a defect analysis line chart that lets us see defect
removal process patterns and understand defect management effectiveness
www.qaexperts.pro

Created Vs Resolved - Cumulative
All Bugs
Difference should be minimal
and If it is big and no positive
trend is shown then probably
Bagathon sprint should be
planned
Jira “Created vs. Resolved Chart” Gadget
www.qaexperts.pro

Defects Dept
Critical and Blocker Bugs
Note: Should Be always close to Zero for Critical bugs
All Bugs
Spreadsheets + advanced Jira time period based queries
Defects dept is a cumulative difference between Resolved and Open bugs
= (Open in sprint N - Resolved in sprint N)+ (Open in sprint N+1- Resolved in sprint N+1 )+…
Note: Shows accumulation of bugs in time
www.qaexperts.pro

Bugs Opened vs Stories Delivered
There should be correlation between the metrics - more stories you deliver more bugs may appear.
Ideal trend is to deliver more user stories with less bugs.
!!!Pay attention to anomalies and investigate them deeper.
Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro

Bugs per Story Point
Critical
and Blocker Bugs
If number
decreases in
time it means
that we
improve QA
process Note: This metric is very good for cases when team size or
product increment is variable or change (increase)
All Bugs
Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro

Bugs per Story Point
Other Variations:
1. Bugs per requirement
2. Bugs per developer
3. Bugs per 1000/Code
4. Time lost for bugs h - Time spent for Bugs vs time spent for stories
Note: Very useful to for making future project estimations.
www.qaexperts.pro

Fix Rate
Should be close to 100% or higher
Spreadsheets + Advanced Jira period based queries
Critical and Blocker Bugs
All Bugs
Fix Rate = Closed bugs / Open bugs
Ideally should be close to 100%
www.qaexperts.pro

“Average Age” and “Time in status”
For Critical
and Blocker
Bugs
Jira “Average Age” and “Time in Status” Gadgets
Average Age - Displays the average number of days issues have been unresolved.
For critical and Blocker bugs this time should be very short.
If process improves it should decrease in time.
Average Time in status, e.g. Open status
- Displays the average number of days issues have spent in status.
For critical and Blocker bugs this time should be very short.
If process improves it should decrease in time.
www.qaexperts.pro

Cumulative Flow Chart
Jira Agile Reports
To find process bottlenecks and predict project close time
www.qaexperts.pro

Bugs by Components/Platforms
Pivot or other spreadsheet Report
Bugs trend By Component – Critical and Blocker / All Bugs
Shows weakest component of the product, shows
where additional testing should be performed, shows
where technical Improvements should be made,
Shows whether component quality improves
Bugs trend By Platform – Critical and Blocker / All Bugs.
Shows amount of cross-platform bugs, shows the
weakest platform requiring more testing and
stabilization.
.
www.qaexperts.pro

Sprint and Business Metrics
Estimates Accuracy % = Estimated amount of
effort/Actual amount of effort-1
Story Delivery Rate % = Actual Stories Completed /
Committed Stories
% of Change Requests = Number of change requests /
Number of Initial Backlog stories
Change Requests Costs = Time spent of change
requests * Average Rate of developers on the project
% Costs of Defect = Average Time for defect fix *
Average developer rate on the project
 Trends
 For Sprint
 General value
www.qaexperts.pro

User Satisfaction
Number of hot fixes had to be made in this or that release.
Shows leaks of Critical bugs on prod.
Bugs found by end-users
Critical/Blocker (Shows leaks of Critical bugs on prod.)
Major and Low priority
Customer Complaints and Feedbacks reports. (Net Promoter Score)
A/B Testing results
Shows which functionality is more preferred
Other Marketing investigations and surveys
All these answer the following questions:
a) What are the weakest points of the product?
b) What functionality is most used ?
c) Which focus in product development and testing should be made?
www.qaexperts.pro

Qualitative Metrics about the project state
If you want to gather more metrics about project processes, approaches,
technology, communication, mood in teams, etc.
Just ask people to provide evaluation and feedback
Experts evaluation and/or Audit
Surveys
Retrospectives
1-2-1 and other meeting
Rumors/small talks at Smoking room/kitchen 
www.qaexperts.pro

QA Metrics
for
Team and
Individual
Efficiency
analysis
www.qaexperts.pro

Developers and Dev Team efficiency
REOPENED:
Reopened Bugs Ratio % = Reopened bugs vs Resolved bugs
Reopened Bugs Ratio % = Reopened Stories vs Stories
delivered to testing
STORY POINTS DELIVERED
STORY DEFECTS ISSUED
STORY DEFECTs/STORY POINTs rate %
QUALITATIVE METRICS: Gather qualitative metrics about
developers: approach to coding, complexity of code,
contribution to innovation, communication efficiency, quick
reaction to problems, etc.  Per PROJECT in general
 Per Team
 Per specific Developer
www.qaexperts.pro

QAs and QA team Efficiency
REJECTED
Absolute Number of rejected bugs:
All resolutions,
Not a bug,
Duplicate
Cannot reproduce
Incomplete
Rejected Rate % = rejected vs created
BUGs LEAKS: to PROD other later stages of development.
Break down by
Per product part
Per functionality
So you can correlate it with a certain QA or group of QAs
Separately:
 For critical and blocker
 For Lower priority
defects
 Per PROJECT in general
 Per QA Team
 Per specific QA
www.qaexperts.pro

QAs and QA team Efficiency
WORK DONE:
 Stories Tested
 Created bugs Critical and Blocker (not rejected)
 Created bugs Major and lower priority (not Rejected)
 Test cases Created (passed DoD). Test cases rejected.
 Test cases Executed.
 Automation scripts Created (passed DoD). Test cases
rejected.
 Advanced test artifacts created and activities
accomplished: test suites, plans, release support, etc.
QUALITATIVE METRICS: Gather qualitative metrics about QAs:
Complexity of tasks, Communication efficiency, Proactivity,
Contribution to innovation, Test management skils, Conflict solving,
etc.
 Per QA team
 Per specific QA
 For Sprint (Velocity)
 Trends
 General value
www.qaexperts.pro

QA Metrics for
Test
Automation
www.qaexperts.pro

 Unit test coverage & number of test
 API testing coverage & number of test
for product in general
by component
 GUI Automation testing coverage & number of test
 for product in general
 by product part
 by platform/device
 by type of test
end-to-end test
Smoke test
Regression test
UAT test
 by test priority:
Priority 1
Priority 2
Priority 3
Automation Coverage & Results
www.qaexperts.pro

Automation Efficiency
AUTOMATION SPEED:
 Automation tests Frequency
 Duration of tests Running
 Velocity of script creation and maintenance
 Time for script creation and maintenance
AUTOMATION EFFICIENCY:
Stability of automation
False Positives
Root Cause for failure
Bugs found by Automation
Critical and Blocker
Other bugs
Bugs missed with automation.
Critical and Blocker
Other bugs
ROI – Time automation economy, etc.
Cost per automated test
 Per QA team
 Per specific QA
 For Sprint (Velocity)
 Trends
 General value
www.qaexperts.pro

QA Metrics
TOOLS
www.qaexperts.pro

Tools
 Jira + Special time-
related JQL
 Confluence + Jira and
other Macros in it
 Spreadsheet processors:
Googlesheets, Excel
 TestTrail
www.qaexperts.pro

JiraReportingInGsheets Tool
Link to test version:
https://chrome.google.com/webstore/detail/jirareportingingsheets/jmjoohlbcigmigokjldgcgnfofcagock
JiraReportingInGSheets add-on
for GoogleSheets
allows to automatically build custom jira
reports in google sheets
Created by QAExperts.pro
www.qaexperts.pro

Thank you
Darya Alymova and Sergey Sosnitskiy
for their contribution to certain metrics definition and
implementation
www.qaexperts.pro

QA Experts.Pro Workshops
2 days workshop
November 2017
http://qaexperts.pro/#signup
www.qaexperts.pro

QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных сторон. Реальное применение

Recommended

More Related Content

What's hot (19)

Similar to QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных сторон. Реальное применение (20)

More from QAFest (20)

Recently uploaded (20)

QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных сторон. Реальное применение

Editor's Notes