2.6 Property-Based Testing

We briefly mentioned property checks when talking about design-by-contracts. There, we used properties in assertions. We can also use properties for test case generation instead of just assertions.

Given that these properties should always hold, we can test them with any input that we like. We typically make use of a generator to try a large number of different inputs, without the need of writing them all ourselves.

These generators often create a series of random input values for a test function. The test function then checks if the property holds using an assertion. For each of the generated input values, this assertion is checked. If we find an input value that makes the assertion to fail, we can affirm that the property does not hold.

The first implementation of this idea was called QuickCheck and was originally developed for Haskell. Nowadays, most languages have an implementation of quick check, including Java. The Java implementation we are going to use is jqwik.

Getting started with property-based tests

How does it work?

• First, we define properties. Similar to defining test methods, we use an annotation on a method with an assertion to define a property: @Property. QuickCheck includes a number of generators for various types. For example, Strings, Integers, Lists, Dates, etc.

• To generate values, we add some parameters to the annotated method. The arguments for these parameters will then be automatically generated by jqwik. Note that the existing generators are often not enough when we want to test one of our own classes; in these cases, we can create a custom generator which generates random values for this class.

• jqwik handles the number of generated inputs. After all, generating random values for the test input is tricky: the generator might create too much data to efficiently handle while testing.

• Finally, as soon as jqwik finds a value that breaks the property, it starts the shrinking process. Using random input values can result in very large inputs. For example, lists that are very long or strings with a lot of characters. These inputs can be very hard to debug. Smaller inputs are preferable when it comes to testing. When an input makes a property fail, jqwik tries to find a shrunken version of this input that still makes the property fail. That way it gets the smallest part of a larger input that actually causes the problem.

As an example: a property of Strings is that if we add two strings together, the length of the result should be the same as the sum of the lengths of the two strings summed. We can use property-based testing and jqwik's implementation to make tests for this property.

public class PropertyTest {

  @Property
  void concatenationLength(@ForAll String s1, @ForAll String s2) {
    String s3 = s1 + s2;

    Assertions.assertEquals(s1.length() + s2.length(), s3.length());
  }
}

concatenationLength had the Property annotation, so QuickCheck will generate random values for s1 and s2 and execute the test with those values.

Property-based testing changes the way we automate our tests. We have only been automating the execution of our tests; the design and instantiation of test cases were always done by us, testers. With property-based testing, by means of QuickCheck's implementation, we also automatically generate the inputs of the tests.

Other examples

Suppose the following requirement:

Requirement: Passing grade

A student passes an exam if s/he gets a grade >= 5.0. Grades below that are a fail.

Grades range from [1.0, 10.0].

We can identify two valid classes and one invalid class in the requirement: passing grades and non-passing grades, and grades outside the range.

When doing property-based testing, we declare these properties in form of jqwik's properties:

• The fail property: for all floats, ranging from 1 (inclusive) to 5.0 (exclusive), the program should return false.
• The pass property: for all floats, ranging from 5 (inclusive) to 10 (inclusive), the program should return true.
• The invalid property: for all invalid grades (which we define as any number below 0.9 or greater than 10.1), the program must throw an exception.

public class PassingGradesPBTest {

    private final PassingGrade pg = new PassingGrade();

    @Property
    void fail(@ForAll @FloatRange(min = 1f, max = 5.0f, maxIncluded = false) float grade) {
        assertThat(pg.passed(grade)).isFalse();
    }

    @Property
    void pass(@ForAll @FloatRange(min = 5.0f, max = 10.0f, maxIncluded = true) float grade) {
        assertThat(pg.passed(grade)).isTrue();
    }

    @Property
    void invalid(@ForAll("invalidGrades") float grade) {
        assertThatThrownBy(() -> {
            pg.passed(grade);
        })
        .isInstanceOf(IllegalArgumentException.class);
    }

    @Provide
    private Arbitrary<Float> invalidGrades() {
        return Arbitraries.oneOf(
                Arbitraries.floats().lessOrEqual(0.9f),
                Arbitraries.floats().greaterOrEqual(10.1f));
    }
}

See another requirement, also used as an example in jqwik's website:

Requirement: FizzBuzz

The program must return 'Fizz' to multiples of 3, 'Buzz' to multiples of 5, and 'FizzBuzz' to multiples of 3 and 5. The program must throw an exception for numbers below 0 (inclusive).

We can derive four clear properties:

• Property fizz: for all numbers divisible by 3, and not divisible by 5, the program returns "Fizz" (see the divisibleBy3 provider method to understand how we feed values with such properties).
• Property buzz: for all numbers divisible by 5 (and not divisible by 3), the program returns "Buzz".
• Property fizzbuzz: for all numbers divisible by 3 and 5, the program returns "FizzBuzz".
• Property noZeroesAndNegatives: the program throws an exception for all numbers that are zero or smaller.

public class FizzBuzzTest {

    private final FizzBuzz fb = new FizzBuzz();

    @Property
    boolean fizz(@ForAll("divisibleBy3") int i) {
        return fb.fizzbuzz(i).equals("Fizz");
    }

    @Property
    boolean buzz(@ForAll("divisibleBy5") int i) {
        return fb.fizzbuzz(i).equals("Buzz");
    }

    @Property
    boolean fizzbuzz(@ForAll("divisibleBy3and5") int i) {
        return fb.fizzbuzz(i).equals("FizzBuzz");
    }

    @Property
    void noZeroesAndNegatives(@ForAll("negative") int i) {
        assertThrows(IllegalArgumentException.class, () -> fb.fizzbuzz(i).equals("FizzBuzz"));
    }

    @Provide
    Arbitrary<Integer> divisibleBy3() {
        return Arbitraries.integers()
                .greaterOrEqual(1)
                .filter(i -> i % 3 == 0)
                .filter(i -> i % 5 != 0);
    }

    @Provide
    Arbitrary<Integer> divisibleBy5() {
        return Arbitraries.integers()
                .greaterOrEqual(1)
                .filter(i -> i % 5 == 0)
                .filter(i -> i % 3 != 0);
    }

    @Provide
    Arbitrary<Integer> divisibleBy3and5() {
        return Arbitraries.integers()
                .greaterOrEqual(1)
                .filter(i -> i % 3 == 0)
                .filter(i -> i % 5 == 0);
    }

    @Provide
    Arbitrary<Integer> negative() {
        return Arbitraries.integers().lessOrEqual(0);
    }
}

You may see other examples in our code repository.

Exercises

1. Write property-based tests for all the exercises we discussed in the domain testing chapters and appendix.