Simple Measurements #2

Simple Measurements
Schalk W. Cronjé
ysb33r @ gmail.com
@ysb33r

CT SPIN 2012 © Schalk W. Cronjé

What is Agile?


“What is Agile?” - Responses

● “Continuous development done in small iterations”

● “Adapting to rapid change, predictable periodic
delivery of usable software”

● “Developers and testers work together from
beginning to end”

● “You'd only get a cynical answer from me”


Being an Agile Fundamentalist
● Increased revenue
● Reduction in costs
● Faster response times
● Less rework

Agile is the continuous delivering of value
to stakeholders
● Customers
● Users
● Developers & Testers
● Governments & Legislators
● Societal Institutions

"The engineering method is the use of
heuristics to cause the best change in a
poorly understood situation within the
available resources."

- Billy V Koen


Accuracy is to precision
as
engineering is to mathematics


Accuracy is to precision
as
engineering is to mathematics

Be accurate enough
to get there, but no more accurate


Software Engineering

● Implies measurement
● Implies Kolb / Shewhart / Deming Cycles
● Is not code craftsmanship
● SE receives bad press
– Perceived difficulty in measurement
– Lack of measurement in S/W dev


Simple Measurements:

A foundation of software engineering,
NOT
a tool for measuring humans


Why are you not measuring
your process today?


How do you know when your
delivery was successful?


Signature Skills

● Three parts
– Preferred Task
– Preferred Cognitive Approach
– Preferred Technology
● This may affect how/what you measure!
● Combat with
– Appropriate team composition
– Reviews / quality measures


Batch Development

Feature 1 Dev Feature 1 QA

delivered
build


Batch Development
Total Time = devt1 .. devt4 + qat1 …qat4 + fixdelay + fixtime + retest time


delivered
build

defect trickle feed

Bug Feature 5+ QA
DB Retest fixes QA
delivered
build
Feature 5+ Dev


Time Factors
Building test
understanding infrastructure
specs
basic build
verification
delivered
build

time from raising defect until it is
available for testing
defect trickle feed time to
re-test

Bug Feature 5+ QA
DB Retest fixes QA
delivered
build
Feature 5+ Dev


Taking an Economic View

● It helps to quantify the effects of multiple interacting
variables
● It helps us to understand that the customer is not the only
judge of value
● By using an economic framework it can allow us to
maximise value, including
– cycle time
– product cost
– development expense
● It helps to communicate with non-technical decision
makers


"If you only quantify one thing,
quantify the cost of delay."

- Don Reinertsen


Cost of Delay – Quality Issue

● One small feature, not properly tested
● Found post-release
● Time wasting by users having to manual work
quantified => £35,000
TotalManualTimeSpent x AvgCostToCompanyPerUser +
TimeToRework x AvgCostToCompanyPerDev


"A pilot who sees from afar
will not make his boat a wreck."

- Amen-en-apt
(Egyptian philosopher, ~700BC)


David Anderson's Six Steps

● Focus on quality
● Reduce Work-in-Progress
● Deliver often
● Balance demand against throughput
● Prioritise
● Attack sources of variability to improve
predictability


Business Value Increments

● Delivery is in business value increments
(BVIs)
● Only “done” when the increment is delivered
end-to-end
● No burning down of tasks ! (false economy)
● Each increment must have at least one metric
of the value it will deliver


Properties of a Kanban System


Visualisation

● Honesty!
● Makes it transparent to everyone that is going
on
● Visual cue for bottlenecks
● Manual board and easy-to-use online system


Visualisation


Limit Work-in-Progress
R e a d y S p e c S p e c D e v D e v Q A Q A R e le a s e d
C o m p le te C o m p le te C o m p l e te

● Use a pull system
● Managing queues are easier than managing time lines
● Allows for better throughput utilisation
● Identifies bottlenecks in the system
● Measuring cycle time allows better prediction than traditional
estimation


Policies
R e a d y S p e c S p e c D e v D e v Q A Q A R e le a s e d
C o m p le te C o m p le te C o m p l e te

● Each stage has a policy
● Describes generic activities required
● “Definition of Done” for each stage


Feedback & Batch Size

● Faster feedback makes learning faster and
more efficient
● Faster feedback provides a sense of control
● Large batches leads to slower feedback


Measuring Cycle Time

● Measure average time in system of each BVI
● Less time is spent on estimating future delivery times
● Historic data automatically takes into account any
disruptions such as
– Operational issues
– Days off sick

AvgTimeInSystem x NoOfBVIs
Time = ---------------------------------
Concurrency


Historic Data vs Estimation


Cadences

● Delivery is done at regular cadences.
● What is ready, is delivered, what is not, is left
out
● No artificial time-boxing or break points as in
SCRUM-like approaches
● Cycle-time leads to better predictability of what
can actually be delivered


Case study

3 Teams – 3 Locations – 18 months


Simple Flow Model
Specify Develop QA

Kanban
Board
Ready Spec Spec Dev Dev Q A Q A R e le a s e d
C o m p l e te C o m p le te C o m p le te


Reduced Learning Time
● Writing test specifications after the
development is costly in time
– Knowledge decay
– Leads to incomplete designs
● Move the test specification up-front before
any development starts
– This is part of requirements discovery
– It broadens the perspective of the programmer
– Allows us to distinguish between automated checks
and human testing


Reduced Learning Time
Specify Develop QA

Test specification

+Time to specify -Time to re-work

+Time to develop -Time to re-test


Changed the Flow Model
Specify Develop & Check Verify

● Ensure the all automated checks are included
in the development stage
● Free up a Verification stage for exploratory
testing and validating that process has been
fulfilled.


Real-world measurements

S1 S2 S3 S4
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / 3 6 4 18
Completed
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42

Average cycle days per feature


Pathology

S1 S2 S3 S4
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
DEV + Check / 3 6 4 18
Completed
VERIFY 3 7 8 1
Total 17 26 43 42



Pathology
Excessive long time between feature availability and testing. Is there enough
testing bandwidth? Or is this team just cheating?

S1 S2 S3 S4
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
DEV + Check / 3 6 4 18
Completed
VERIFY 3 7 8 1
Total 17 26 43 42



Kanban fails when people
don’t want to face the truth.

- Hillel Glazer


It all fell apart ...

● Restructure – 3 teams + 3 managers
● 2 managers rejected Kanban
– 1 manager wanted ScrumWorks
– 1 manager used Microsoft Project
● 1 manager was forced not to use Kanban
– Switched to iteration-based deliveries
● All teams forced back to time-based
estimation


It fell apart even more ...

● 1 team decided on 4 wks dev + 4 wks testing
– Quality was worse than before
– Cycle time was at least 8 calendar weeks per BVI

● 1 team did not deliver for over 5 months
– Finally delivered something the user did not want

● None of teams have any measurements on
cycle time; cannot predict how long delivery
will take


What did the team members
say?
● “I have lost visualisation. I have no idea what is going
on”

● “We are not using Kanban, because the company forced
us to SCRUM”

● “If I was given a choice, my the team and I would
definitely go with Kanban”

● “It is not the company 'standard' as such you need to
'sell' the process frequently. “


Quantifying BVIs


Quantifying BVIs

● Know what you want to achieve
● Establish a scale for measurement
● Decide how to measure
● Make constraints explicit
● Communicate


Quantify BVIs - Example

For all customer submissions that meets
the following criteria, 80% should be
handled by an automation system instead
of a human and resolution achieved within
30 minutes.

Criteria A
Criteria B
etc.


Planguage

● Created by Tom Gilb
● Provides a structured way of quantifying
requirements
● Allows for reuse of specifications


Quantify BVI - Planguage

Name: Customer submissions resolved by automation systems

Scale: Percentage of submissions resolved by automation in
relation to all submissions

Meter: Query on database using the following statement:
SELECT … FRO M… WHERE . . .

Goal: 80% [Criteria A, Criteria B]


Quantified BVI - Applying the
Economic Model

● Compare to average human time to respond
and volume covered
● Extrapolate to time savings or cost savings
● This allows for two measures of success
– Technical
– Financial


End game


Apply Lean Principles
to your Team

● Start from where you are; model the workflow
● Make the system pull-based
● Reduce batch size; limit the work-in-progress
● One feature end-to-end
● If the flow breaks, fix it immediately


to your Team
● Pair up people with different skills to work on
same feature
– Feature teams are not a new concept
– Reduces mind-set trap, creative abrasion
● Could be perceived to have higher person
cost per feature
– Instead of distributing the person cost over
multiple queues, the cost is combined into a
single queue
– Can actually lead to reduced cycle time.

Simple Measurements

● Measure cycle time (now predict delivery)
● Quantify requirements (know that you did deliver)
● Use “Cost of Delay” (prioritise, show impact/risk)


Thank you


Simple Measurements #2

More Related Content

What's hot (20)

Similar to Simple Measurements #2 (20)

More from Schalk Cronjé (20)

Recently uploaded (20)

Simple Measurements #2

Editor's Notes