Property-based testing

Property-based Testing
for Everyone
Dmitriy Morozov / Zoomdata Tech Lunch

Example-based
Testing
• set up one input scenario
• run the code under test
• check the output 
(and any other effects the code is supposed to have)

Imagine we needed to
implement a custom ADD
function

@Test 
public void testSum() { 
assertThat(sum(1, 2), equalTo(3)); 
}

private int sum(int a, int b) { 
if (a == 1 && b == 2) { 
return 3; 
} else { 
return 0; 
} 
}

@Test 
public void testSum() { 
assertThat(sum(2, 2), equalTo(4)); 
}

if (a == 1 && b == 2) { 
return 3; 
} else if (a == 2 && b == 2) { 
return 4; 
} else { 
return 0; 
} 
}

@Test 
public void testSum3() { 
// When I add two numbers I expect their sum 
for (int i = 0; i < 100; i++) { 
int x = randomInt(); 
int y = randomInt(); 
assertThat(sum(x, y), equalTo(x + y)); 
} 
}

@Test 
public void testSumUsingCommutativityProperty() { 
// Changing the order of arguments
// shouldn’t change their sum  
for (int i = 0; i < 100; i++) { 
int y = randomInt(); 
assertThat(sum(x, y), equalTo(sum(y, x))); 
} 
}

// unfortunately that's not enough 
 
return a * b; 
}

@Test 
// `x + 2` is the same as `x + 1 + 1`  
public void testSumWithAdditiveProperty() { 
for (int i = 0; i < 100; i++) { 
int result1 = sum(sum(x , 1), 1); 
int result2 = sum(x, 2); 
assertEquals(result1, result2); 
} 
}

// WTF?!
return 0; 
}

@Test 
// `x + 1` = `x` 
public void testSumWithIdentityProperty() { 
for (int i = 0; i < 100; i++) { 
assertEquals(x, sum(x, 0)); 
} 
}

Testing Speciﬁcation with properties
• Commutativity property
• Associativity property
• Identity property

Testing Speciﬁcation with properties
By using speciﬁcations, we have understood the
requirements in a deeper way

John Hughes - Testing the Hard Stuff and Staying Sane

Beneﬁts
• Less time spent writing test code 
- One property replaces many tests
• Better testing 
- Lots of combinations you’d never test by hand
• Less time spent on diagnosis 
- Failures minimized automatically

3,000 pages of speciﬁcations
20,000 lines of QuickCheck
1,000,000 LoC from 6 suppliers
200 bugs
100 bugs in the standard

@RunWith(JUnitQuickcheck.class) 
public class AdditionPropertyTest { 
 
@Property 
public void commutativityProperty(int x, int y) { 
assertThat(sum(x, y), equalTo(sum(y, x))); 
} 
 
@Property 
public void additiveProperty(int x) { 
int result1 = sum(sum(x , 1), 1); 
int result2 = sum(x, 2); 
assertEquals(result1, result2); } 
 
@Property 
public void identityProperty(int x) { 
assertEquals(x, sum(x, 0)); 
}
 
}

java.lang.AssertionError: Property identityProperty
falsified via shrinking: expected:<1> but was:<0>
Shrunken args: [1]
Original failure message: [expected:<98> but was:<0>]
Original args: [98]
Seeds: [7329045779044418918]

@RunWith(JUnitQuickcheck.class) 
public class MoneyChangerTest { 
 
@Property 
public void sumOfCoinsEqualsAmount( 
@InRange(min = "0", max = "500") int amountToChange, 
Set<@InRange(min = "1", max = "100") Integer> denominations) { 
 
assumeThat(denominations.size(), greaterThan(0)); 
denominations.add(1); 
 
MoneyChanger moneyChanger = new MoneyChanger(); 
List<Integer> coins = moneyChanger.change(amountToChange, denominations);
 
int sum = coins.stream().mapToInt(coin -> coin).sum(); 
assertThat(sum, equalTo(amountToChange)); 
} 
 
}

describe("sort", function () { 
jsc.property( 
"idempotent", "array nat", function (arr) { 
return _.isEqual(sort(sort(arr)), sort(arr)); 
}); 
});

How to handle state?
• Generate actions changing the state
• Apply each action to the state, if possible
• After each application, verify the model

Where can we beneﬁt
from
property-based tests?

Spark Proxy on DataFrame
API
• Generate random DataReadRequest
• Execute request with RDD-based code
• Execute same request with DataFrame based code
• Compare the results

SDK
• Send START_VIS
• Generate random request, change metric, group-
by, ﬁlter, sort, etc
• Validate responses based on the test model
• Send STOP_VIS

• QuickCheck
• Testing Telecoms Software with Quviq QuickCheck
• Testing the hard stuff and staying sane
• The Mysteries of Dropbox
• Jepsen IV: Hope Springs Eternal
• Java Examples from this talk

Property-based testing

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to Property-based testing (20)

Recently uploaded (20)

Property-based testing