Shift-Left Testing: QA in a DevOps World by David Laulusa

Shift-Left Testing
QA in a DevOps World
David Laulusa
@dlaulusa
Feb 25, 2020

About Me
• Brigham Young University, BS/EE
• Princeton University, MS/EE
• 20+ yrs in industry
• Development/QA
• Startups, Large Enterprises, Permanent/Contract
• Telecommute
• DevOpsDays Columbus

Agenda
Inject quality earlier in the process

You cannot inspect quality
into a product.
- Harold F. Dodge

Inspection does not improve the quality,
nor guarantee quality. Inspection is too
late. The quality, good or bad, is already
in the product.
- William Deming

Quality is not an act.
It is a habit.
- Aristotle

Agenda
• Testing Principles
• Testing Partners
• Test

Continuous Delivery is the NEW Agile
Satisfy the customer through early and
continuous delivery of valuable software
- Agile Manifesto

F1
Design
Dev
Test
Deploy
F2
Design
Dev
Test
Deploy
B1
Design
Dev
Test
Deploy
B2
Design
Dev
Test
Deploy
F3
Design
Dev
Test
Deploy
B3
Design
Dev
Test
Deploy

Shift-Left Testing
Static Unit Integration End-to-End Acceptance

Bugs are cheaper when caught young.
- Larry Smith

Testing a System
System
Expected Outputs
Inputs

Input Domain
• The input domain contains all possible inputs.
• Choose a finite set of values.
• Include valid, invalid and special values
• Include values that represent “normal use” (domain-specific knowledge)
• Test selection is about sampling the input space in a cost-effective manner.

Equivalence Classes
• All members of an equivalence class contribute to the fault detection
in the same way.
• The combination of all equivalence classes must cover the entire
domain.
• Equivalence classes must not overlap.
• Select one value from each Equivalence Class.
• Choosing more than one value does not increase fault detection.

Range
• One class with values inside the range, and two with values outside
the range.
• If the range were 10 <= x <=20.
1.Equivalence Class 1: all numbers between 10 and 20, inclusive
2.Equivalence Class 2: all numbers less than 10
3.Equivalence Class 3: all numbers greater than 20
4.You could also do letters and special characters.

String
• At least one containing all legal strings and one containing all illegal
strings.
• For example, let fname: string be a variable to denote a first name.
1. Valid name {Sue}
2. Special characters {ε}
3. Number {7}
4. Too long {Too many characters so it would not be able to fit into
the database}

Enumeration
• Each value in a separate class.
• And then any other value.
• For example, consider auto_color ∈ {red, green , blue}.
• The following classes are generated, {{red}, {green}, {blue}, {blah}}

Array
• One class containing all legal arrays, one containing only the empty
array, and one containing arrays larger than the expected size.
• For example, consider int[] aName = new int [3].
• The following classes are generated:
1. {[]}
2. {[-10, 20]}
3. {[-9, 0, 12, 15]}

Boundary-Value Analysis
Programmers often make mistakes in processing values at and near
the boundaries of equivalence classes.
• min-1, min, min+1
• max-1, max, max+1
• 0, null, empty string

Combinatorial Testing
• Many defects only occur when a combination of inputs or events
occur that interact with each other.
• However, if you were to test every combination of inputs it may take
years especially with today’s complex systems.
• How do you choose which combination of inputs?
• Combinatorial testing is a technique for reducing the number of test
cases without drastically compromising functional coverage, in order
to get more ‘bang for the buck’.

Testing a System
X (1,2) Y (Q,R) Z (5,6)
System

Combinatorial Testing Where T = 2,
Pairwise Testing
Test Case X Y Z
TC1 1 Q 5
TC2 1 R 5
TC3 1 Q 6
TC4 1 R 6
TC5 2 Q 5
TC6 2 R 5
TC7 2 Q 6
TC8 2 R 6

Test Case X Y Z
TC1 1 Q 5
TC2 1 R 5
TC3 1 Q 6
TC4 1 R 6
TC5 2 Q 5
TC6 2 R 5
TC7 2 Q 6
TC8 2 R 6
Pairwise Testing

Pairwise Testing
Test Case X Y Z
TC1 1 Q 5
TC4 1 R 6
TC6 2 R 5
TC7 2 Q 6
TC8 2 R 6

Pairwise Testing
Test Case X Y Z
TC1 1 Q 5
TC4 1 R 6
TC6 2 R 5
TC7 2 Q 6

Pairwise Testing
Test Case X Y Z
TC1 1 Q 5
TC4 1 R 6
TC6 2 R 5
TC7 2 Q 6
• Going from 8 -> 4 isn’t that big a deal
• Reducing # tests 50% is not that big a deal either
• Reducing by hand is tedious

Android Options
Parameter Values
HardKeyboardHidden No, Undefined, Yes
KeyboardHidden No, Undefined, Yes
Keyboard 12Key, NoKeys, Qwerty, Undefined
NavigationHidden No, Undefined, Yes
Navigation Dpad, NoNav, Trackball, Undefined, Wheel
Orientation Landscape, Portrait, Square, Undefined
ScreenLayout_Long Mask, No, Undefined, Yes
ScreenLayout_Size Large, Mask, Normal, Small, Undefined
TouchScreen Finger, NoTouch, Stylus, Undefined
3*3*4*3*5*4*4*5*4 = 172,800 combinations

Combinatorial Testing to the Rescue
T # Tests % of Total (172k) % Defects Found
2 29 0.02% 76%
3 137 0.08% 95%
4 625 0.4% 97%
5 2532 1.5% 99%
6 9168 5.3% 100%
• The savings as a percentage of exhaustive testing depends on the
number of parameters and the # of values for the parameters.
• With larger systems the savings can be enormous

“There’s an App for that”
• AllPairs (perl)
• PICT (Microsoft)
• ACTS (NIST)

When Not to Use Combinatorial Testing
• When you really do have to test all combinations
• This assumes a uniform distribution across all
combinations.
• Some combinations are more important than others.
• Some combinations are impossible.

T-Way - Effective in Finding Those Pesky Bugs
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6
# Interacting Parameters Leading to Failure
Browser Bugs (194) Server Bugs (171)

Fly With Us – We found 90% of the defects!

Dependency Injection
In software engineering, dependency injection is a software design pattern
that implements inversion of control for resolving dependencies. A
dependency is an object that can be used (a service). An injection is the
passing of a dependency to a dependent object (a client) that would use it.

sub method {
my $self = shift;
my $obj = MyProject::Class->new();
my $x = $obj->get_something();
. . .
return($x);
}

sub method {
my $self = shift;
my $obj = shift;
my $x = $obj->get_something();
. . .
return($x);
}

Grooming the Backlog
• Make sure that there are sane completion requirements
• Is it testable?

Why Do We Unit Test?
static OSStatus
SSLVerifySignedServerKeyExchange(SSLContext *ctx, bool isRsa, SSLBuffer
signedParams, uint8_t *signature, UInt16 signatureLen) {
OSStatus err;
...
if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)
goto fail;
if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;
goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
goto fail;
...
fail:
SSLFreeBuffer(&signedHashes);
SSLFreeBuffer(&hashCtx);
return err;
}

Why Do We Unit Test?
• It gives you confidence that your code works as you expect it to work.
• Easily refactor your code to make it cleaner and more efficient.
• Saves time because unit tests help prevent regressions from being introduced and
released.
• Once a bug is found, you can write a unit test for it, you can fix the bug, and the bug can
never make it to production again because the unit tests will catch it in the future. (shift left
testing)
• Implicit documentation

How Do We Unit Test?
• Unit tests test units in isolation, in a dedicated, controlled environment.
• Each unit test creates in own environment by instantiating only the classes
needed to execute one test, putting them in a known state, then it invokes the
method to be tested and verify the outcome.
• This verification is done by assertions on the behavior of the method (as opposed
to its implementation).
• Performing the verification on the behavior and not on the implementation is
important as this allows modifying the implementation without breaking the unit
tests, and therefore using the unit tests as a safety net for the modification.

What Do We Unit Test?
• Generally a method.
• Need to write tests for the public methods.
• Could write tests for private methods, but it means you might have
to re-write these tests when you re-factor.
• No need to write tests for trivial setter/getters.

Anatomy of a Unit Test
• Set up all conditions for testing.
• Call the method (or Trigger) being tested.
• Verify that the results are correct.
• Clean up modified records.

Unit Tests
• Make each test orthogonal (i.e., independent) to all the others
• Don’t make unnecessary assertions
• Test only one code unit at a time
• Mock out all external services and state
• Avoid unnecessary preconditions
• Unit tests should ideally take less than 5 min to run. If it approaches 10 min, then
we can break them up and run sets in parallel.
• Unit tests will run each time code is checked in.
• Builds cannot progress unless all unit tests pass.

A Test is not a unit test if:
• It talks to the database
• It communicates across the network
• It touches the file system
• It can't run at the same time as any of your other unit tests
• You have to do special things to your environment (such as editing
config files) to run it.

Test-Driven Development
• write a failing test
• write just enough code to make it pass
• refactor the code to clean it up

TDD
• TDD is a design process, not a testing process.
• Unit tests help to shape your design.
• TDD is a robust way of designing software components (“units”) interactively so
that their behavior is specified through unit tests.
• TDD helps you to deliver software components that individually behave according
to your design.
• A suite of good unit tests is immensely valuable: it documents your design, and
makes it easier to refactor and expand your code while retaining a clear overview
of each component’s behavior.

Code Coverage
• Code coverage is a measure used to describe the degree to which
the source code of a program is executed when a particular test
suite runs.
• A program with high code coverage, measured as a percentage,
has had more of its source code executed during testing which
suggests it has a lower chance of containing undetected software
bugs compared to a program with low code coverage.

Code Coverage
• Function coverage
• Statement coverage
• Branch coverage
• Conditional coverage

Code Coverage
sub foo {
my($x, $y) = @_;
if ($x eq ‘string’ && $y < 10) {
return 1;
}
else {
return 0;
}
}

Profiling Your Code
Profiling is a form of dynamic program analysis that can measure
things like:
• memory usage
• time complexity of a program
• usage of particular instructions
• frequency and duration of function calls

Profiling Your Code
• optimization
• algorithm
• loops
• expensive function calls, including system calls
• lazy building

Acceptance Testing - Example
Test Specification:
• Free shipping a month before Christmas

Test Code
@Test
public void
orderLessThenAMonthBeforeChristmasShouldGiveZeroShipping() {
Calendar purchaseDate = GregorianCalendar.getInstance();
purchaseDate.set(Calendar.YEAR, 2014);
purchaseDate.set(Calendar.MONTH, Calendar.NOVEMBER);
purchaseDate.set(Calendar.DAY_OF_MONTH, 24);
int christmasEve = 24;
int expectedShipping = 0;
String expectedCurrency = “USD";
Date purchase = purchaseDate.getTime();
Shipping shipping = new Shipping(purchase, christmasEve);
int actualCost = shipping.getCost();
assertThat(actualCost, is(expectedShipping));
}

Cucumber - Layers
Execute Specification (Gherkin)
Step Definitions - Glue (47
languages)
System Under Test

Cucumber - Specification
Scenario: Free shipping a month before Christmas
Given a customer that Christmas is celebrated 24 of December
When a customer buys a book on 2014-12-10
Then the shipping cost should be $0

Cucumber - Step Definitions
@When("^a customer buys a book on (.*)$")
public void
buyBook(@Format("yyyy-MM-dd") Date purchaseDate)
throws Throwable {
this.purchaseDate = purchaseDate;
}

Mutation Testing
• White Box Testing
• Mutate (change) certain statements in the source code and check if
the test cases are able to find the errors.
• This is called killing the mutant. Test suites are measured by the
percentage of mutants that they kill.
• New tests can be designed to kill additional mutants.

Other Types of Testing
• Security Testing
• Performance Testing
• Compliance Testing
• Resiliency Testing

PARTNER WITH
EXTERNAL
STAKEHOLDERS

Testing in Production
• Logging
• Server Monitoring
• Application Performance Monitoring
• Health Check

Burger World Burger:
How Many?:
Here: To Go:
Lettuce:
Tomato:
Pickle:
Onions:
Mayo:
Ketchup:
Mustard:
Instructions:
Order Cancel

THANKS
David Laulusa
@dlaulusa

Shift-Left Testing: QA in a DevOps World by David Laulusa

More Related Content

What's hot (20)

Similar to Shift-Left Testing: QA in a DevOps World by David Laulusa (20)

More from QA or the Highway (20)

Recently uploaded (20)

Shift-Left Testing: QA in a DevOps World by David Laulusa