Mutation Testing with PIT (Booster 2014, 2014-MAR-13)

© Computas AS 11.03.14
Mutation Testing with PIT
Filip van Laenen
Booster 2014
2014-03-13

2 © Computas AS 11.03.14
Agenda
• Basics of mutation testing
• Relation to other testing techniques
• Example
• Mutation testing techniques
• Mutation testing tools
• Personal experiences and recommendations

Basics of
Mutation Testing

Mutation Testing in a Nutshell
Seeking The Summoner @ The Daily WTF
http://thedailywtf.com/Articles/Seeking-The-Summoner.aspx

Mutation Testing in a Nutshell (cont'd)

• Unit tests guard the source code
• But who guards the guardians?
• Do the unit tests cover all source code?
• Lines?
• Branches?
• Paths?
• Do the unit tests test the right things?
Mutation testing tests the tests!

int max(int a, int b) {
return (a < b) ? b : a;
}
return (a <= b) ? b : a;
}

“
Does Mutation Testing Work?
In practice, if the software contains a
fault, there will usually be a set of
mutants that can only be killed by a test
case that also detects that fault.
Geist et. al., “Estimation and Enhancement of Real-time
Software Reliability through Mutation Analysis,” 1992

“
Does Mutation Testing Work? (cont'd)
Complex faults are coupled to simple
faults in such a way that a test data set
that detects all simple faults in a program
will detect most complex faults.
K. Wah, “Fault Coupling in Finite Bijective Functions,”
1995

Does Mutation Testing Work? (cont'd)
• “Generated mutants are similar to real faults.”
• Andrews, Briand, Labiche, ICSE 2005
• “Mutation testing is more powerful than
statement or branch coverage.”
• Walsh, Ph.D. Thesis, State University of New York at
Binghampton, 1985
• “Mutation testing is superior to data flow
coverage criteria.”
• Frankl, Weiss, Hu, Journal of Systems and Software,
1997

Relation to Other
Testing Techniques

Relation to Other Testing Techniques
• Unit tests
• Test-Driven Development (TDD)
• Test coverage
• Static code analysis
• Fuzz testing

Relation to Other Testing Techniques

Is Mutation Testing New?
• R. Lipton, “Fault Diagnosis of Computer
Programs,” 1971
• R. Lipton et. al., “Hints on Test Data Selection:
Help for the Practicing Programmer,” 1978
• Historical obstacles:
• No unit testing
• No TDD
• Inefficient implementation
• Hence time-consuming
• No integration with IDEs

Practical Example
of Mutation Testing

Code Along…
https://github.com/computas-fvl/booster2014
git clone
https://github.com/computas-fvl/booster2014.git
mvn clean site:site
org.pitest:pitest-maven:mutationCoverage

Mutation Testing
Techniques

Mutation Testing Techniques
• Three aspects:
• Mutation injection
• Mutation types
• Unit test selection per mutant
• Key properties:
• Efficiency
• Performance

Mutation Injection
• Source code mutation
• Binary code mutation
• Caveats:
• De-mutation
• Compilation errors
• Invalid binary code

Mutation Types
• Some mutations never change behaviour
• Constants reused by unit tests
• Log messages
• Some mutations can change behaviour
• Switching between < and ≠
• Switching between < and ≤
• Some mutations always change behaviour
• Switching between < and ≥

Mutation Types (cont'd)
return (a < b) ? b : a;
}
return (a <= b) ? b : a;
}
return (a >= b) ? b : a;
}

Mutation Types (cont'd)
for (int i = 0; i < 10; i++) …
for (int i = 0; i != 10; i++) …
for (int i = 0; i >= 10; i++) …

Mutation Types Guaranteed to Change
Behaviour
• Negation of the comparison
• Switching between = and ≠
• Switching between < and ≥
• Switching between > and ≤
• Negation of boolean conditions
• Adding a ¬, ! or ~
• Shortcutting boolean conditions
• Replacement with True or False
*

Unit Test Selection
• Goal: find the unit test that “kills” the mutant
• Selection aids:
• Hints
• Name/package matching
• Code coverage tools
• Automatic learning
• Other heuristics

Unit Test Selection (cont'd)
• System:
• 50 classes
• 20 unit tests per class
• 1 ms per unit test
• Unit testing time: 50 × 20 × 1ms = 1s
• 10 mutants per class:
• Brute-force: 10 × 50 × 1s = 6m 20s
• Educated: 10 × 50 × 20 × 1ms = 10s

Unit Test Selection (cont'd)
• System:
• 500 classes
• 20 unit tests per class
• 1 ms per unit test
• Unit testing time: 500 × 20 × 1ms = 10s
• 10 mutants per class:
• Brute-force: 10 × 500 × 10s = 13h 53m 20s
• Educated: 10 × 500 × 20 × 1ms = 1m 40s

Complexity
• f: Number of function points
• φ: Number of function points per class, ≥ 1
• τ: Number of unit tests per function point, ≥ 1
• μ: Number of mutants per function point, ≥ 1
• Brute-force: (f × τ) × (f × μ) = τ × μ × f²
• Educated force: τ × μ × φ × f
• Ideal: τ × μ × f

Complexity (cont'd)
• c: Number of classes
• φ: Number of function points per class, ≥ 1
• τ: Number of unit tests per function point, ≥ 1
• μ: Number of mutants per function point, ≥ 1
• Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c²
• Educated force: τ × μ × φ² × c
• Ideal: τ × μ × φ × c

Loops
for (int i = 0; i < 10; i++) …
for (int i = 0; i < 10; i--) …

Infinite Loops
• Terminate mutants that take too long to run
• What's “too long”?
• Ruins the performance
• Can be hard to predict

Other Problems
• Recursion:
• Stack overflows
• Out of memory exceptions
• Syntax errors
• Segmentation faults

Mutation Testing
Tools

Mutation Testing Tools
• Java:
• PIT
• Jester (Nester, Pester)
• Jumble
• Ruby:
• Mutant
• Heckle
• C#:
• NinjaTurtles

PIT
• Java
• Junit & TestNG
• Maven or command-line
• Operates on byte code
• Large set of mutators
• Also possibly equivalent mutators available
• Highly configurable
• Sensible defaults
• http://pitest.org/

PIT Mutators
• Conditionals Boundary
• Negate Conditionals
• Remove Conditionals*
• Math
• Increments
• Invert Negatives
• Inline Constant*
• Return Values
• Void Method Calls
• Non Void Method Calls*
• Constructor Calls*

PIT Sample Report

Jester
• Java
• JUnit
• Usually run from the command-line
• Grester for Maven2
• Operates on source code
• Runs all unit tests on all classes
• Reporting and documentation could be better
• http://jester.sourceforge.net/
• http://sourceforge.net/projects/grester/

Jester Sample Report Overview

Jester Sample Detailed Report

Pester and Nester
• Pester
• Jester for Python
• PyUnit
• Nester
• Port of Jester for C#
• NUnit
• Integrated with Visual Studio
• But outdated…
• http://nester.sourceforge.net/

Nester Sample Report

Jumble
• Java
• JUnit
• Run from the command-line
• Operates on byte code
• Runs unit tests on a class
• Reporting and documentation could be better
• Claims to be faster than Jester
• http://jumble.sourceforge.net/index.html

Jumble Sample Report
Mutating Foo
Tests: FooTest
Mutation points = 12, unit test time limit 2.02s
..
M FAIL: Foo:31: negated conditional
M FAIL: Foo:34: - -> +
......
Score: 67%

Mutant
• Ruby 1.9 and 2.0
• rspec2
• Good to-the-point reporting
• Good performance
• OK documentation
• Active project
• https://github.com/mbj/mutant

Heckle
• Ruby 1.8
• Doesn't work on Ruby 1.9
• Test::Unit and rSpec
• Runs a set of unit tests on a class or a method
• Good to-the-point reporting
• Good performance
• Virtually no documentation
• http://rubyforge.org/projects/seattlerb/
• http://docs.seattlerb.org/heckle/

Heckle Mutations
• Booleans
• Numbers
• Strings
• Symbols
• Ranges
• Regexes
• Branches (if, while, unless, until)

NinjaTurtles
• .Net:
• C#
• Visual Basic/VB.NET
• Any other .NET language code
• Doesn't seem much alive
• http://www.mutation-testing.net/

NinjaTurtles Mutations
• Sequence point deletion
• Arithmetic operator substitution (*, /, +, -, %)
• Bitwise operator substitution (&, |, ^)
• Branch substituion (condition, always and never
branch)
• Conditional boundary substition (< and <=, >
and >=)
• Substitution of reads from variables,
parameters and fields of the same type
• Substitution of writes to variables of the same
type

Personal Experiences
and Recommendations

Experiences and Recommendations
• Use mutation testing from day 1
• Start on a small code base
• Keep number of unit tests per class low
• Have small classes
• Select a good tool
• Configurable
• Flexible
• One that can output the mutant

Experiences and Recommendations
(cont'd)
• Believe the tool
• Or try to proof that the tool is wrong
• Fix the problem
• Don't turn mutation testing off
• Embrace the “more than 100%” test coverage
• Path coverage
• Less code
• More unit tests
• More intelligent unit tests

Improvements

Improvements
• Integration with more unit testing frameworks
• Better unit test–source code mapping
• Better heuristics
• Parallellisation
• Better reporting
• IDE integration
• Building tool integration

Questions?
Computas AS
Lysaker Torg 45, pb 482
N-1327 Lysaker
NORWAY
Tel +47-67 83 10 00
Fax +47-67 83 10 01
Org.nr: NO 986 352 325 MVA
www.computas.com
Contact:
fvl@computas.com @filipvanlaenen

Mutation Testing with PIT (Booster 2014, 2014-MAR-13)

More Related Content

What's hot (18)

Viewers also liked (19)

Similar to Mutation Testing with PIT (Booster 2014, 2014-MAR-13) (20)

More from Filip Van Laenen (18)

Recently uploaded (20)

Mutation Testing with PIT (Booster 2014, 2014-MAR-13)