Domain testing examples

This chapter follows a problem-based approach. We first show a program requirement, and then, show how we would apply equivalent class analysis and boundary testing.

We will follow a common strategy when applying domain testing, highly influenced by how Kaner et al. do:

1. We read the requirement
2. We identify the input and output variables in play, together with their types, and their ranges.
3. We identify the dependencies (or independence) among input variables, and how input variables influence the output variable.
4. We perform equivalent class analysis (valid and invalid classes).
5. We explore the boundaries of these classes.
6. We think of a strategy to derive test cases, focusing on minimizing the costs while maximizing fault detection capability.
7. We generate a set of test cases that should be executed against the system under test.

See the videos for detailed explanations. See also the JUnit test cases we implemented for them in https://github.com/sttp-book/code-examples/tree/master/src/test/java/tudelft/domain.

Exercise 1: The Sum Of Integers

A program receives two numbers and returns the sum of these two integers. Numbers are between 1 inclusive and 99 inclusive.

Dependency among variables:

• X and Y are independent (X doesn't influence the range of Y, and vice-versa).
• X and Y are used to calculate Sum.

Equivalence partitioning / Boundary analysis

Strategy

• Variables are independent (X does not affect the range of Y, and vice-versa).
• 7 tests for X (3 partitions + 4 boundaries), in point for Y.
• 7 tests for Y (3 partitions + 4 boundaries), in points for X.
• Total of 14 tests

In-point X = 50, In-point Y = 50.

Questions:

• Do we need T3 and T7? Or are they testing the same thing (i.e. > 99)? (Same applies to T10 and T14.) If we remove one of them, we end up with 12 tests.
• What would change if the requirement had something like sum < 167 ?

Exercise 2: The Sum Of Integers, part 2

A program receives two numbers and returns the sum of these two integers. Numbers are between 1 inclusive and 99 inclusive. Final sum should be <= 165.

Dependency among variables:

• X and Y are independent (X doesn't influence the range of Y, and vice-versa).
• X and Y are used to calculate Sum.
• X + Y <= 165

Equivalence partitioning / Boundary analysis

Strategy

• Variables are independent (X does not affect the range of Y, and vice-versa).
• 7 tests for X (3 partitions + 4 boundaries), in point for Y,
• 7 tests for Y (3 partitions + 4 boundaries), in points for X.
• 2 tests for the boundary on Sum.
• Total of 16 tests

In-point X = 50 (taking into consideration that X <= 165 - Y), In-point Y = 50 (taking into consideration that Y <= 165 - X).

Exercise 3: Passing Grade

A student passes an exam if s/he gets a grade >= 5.0. Grades below that are a fail.

Grades range from [1.0, 10.0]. Assume the system doesn't allow for invalid grades (e.g., 0.9, 10.5).

Variables

Dependencies among variables

• No dependencies among input variables (grade is the only one).
• Grade is used to decide the pass/fail.

Equivalence partitioning / Boundary analysis

We do not need to test 0.9 and 10.1 because we assume that the system doesn't allow for them.

Strategy

Boundaries seem to be enough.

Exercise 4: Passing Concepts

The final grade of a student is calculated as follows:

• 1 <= grade < 5 => F
• 5 <= grade < 6 => E
• 6 <= grade < 7 => D
• 7 <= grade < 8 => C
• 8 <= grade < 9 => B
• 9 <= grade <= 10 => A

The system does not allow for invalid grades (e.g. 0.9, 10.5)

Variables

Dependency among variables

*There are no dependencies among the input variables, since we only have one variable.

• The grade is used to decide the concept.

Equivalence Partitioning/Boundary Analysis

Strategy

Test all boundaries, yielding 12 tests.

Exercise 5: The MSc admission problem

A student can only join the MSc if :

• ACT = 36 and GPA >= 3.5
• ACT >= 35 and GPA >= 3.6
• ACT >= 34 and GPA >= 3.7
• ACT >= 33 and GPA >= 3.8
• ACT >= 32 and GPA >= 3.9
• ACT >= 31 and GPA = 4.0 ACT is an integer between 0 and 36 (inclusive). GPA are float variables between 0.0 and 4.0 (single decimal digit) (inclusive).

Variables

Dependency among variables

• ACT and GPA have a joint effect
• ACT and GPA are used to calculate Decision.

Equivalence partitioning / Boundary Analysis

The conditions from the requirements are really close with each other. Because of that, finding values that cross the boundary between partitions is challenging. We also note that the boundary that matters here is from (student being approved) -> (student not being approved). So, for each partition, we should find its boundary that would force the student not being approved.

These boundary points can found through the following process. We first find the on-point of the boundary, for instance (35, 3.6), then discover the closest inputs that would make the student to fail. In this example, (34, 3.6) and (35, 3.5). Note that there exists one special case (= 36, >= 3.5). Since it contains an equality condition, we have two off-points to explore.

Also note that we are using in-points that are on-points too. This is less common, but in the case of this problem choosing another in-point might make the final outcome to still not change between the boundary tests we devised. For example, if we choose a GPA in-point of 3.6, the 35 and 36 ACT will have the same outcome value (as the next rule will intervene).

Strategy

• There are 24 boundaries (for conditions on valid inputs), but some are repeated. 14 boundary tests.
  • (37, 3.5) is an invalid path, so we can ignore. Therefore 13 test cases.

Exercise 6: The printing label

A printer prints mailing labels. The first line is the name of the person.

The program builds the name from three fields: first name, middle name, and last name. Each field can hold up to 30 characters. The label can be up to 70 characters wide.

Variables

Dependencies among variables

FN + MN + LN <= 68

(The difference of 2 (to 70) happens as the system needs to add an empty space in between names)

Equivalence partitioning / Boundary analysis

Strategy

• Each variable has 2 partitions, plus a restriction.

• Let's not combine "invalid strings" with them all. So: 3 tests for exceptional cases + 5 5 5 = 125 + 3 = 128.

• If we focus on the on-points and off-points, and in-points for others, we'd have 4 + 4 + 4 = 12 tests plus invalid cases: 15 tests.

• In-points always taking the FN + MN + LN <= 68 restriction into account.

• Two tests for the extra restriction: 17 tests.

Notes:

• We simplified the output by basically returning "valid" or "invalid". You might also wanna check the final name that was generated.
• Test cases with strings of length -1 might then not be possible.

Exercise 7: Tax Income

Your income is taxed as follows:*

• 0 <= Income < 22100 → Tax = 0.15 x Income
• 22100 <= Income < 53500 → Tax = 3315 + 0.28 * (Income - 22100)
• 53500 <= Income < 115000 → Tax = 12107 + 0.31 * (Income - 53500)
• 115000 <= Income < 250000 → Tax = 31172 + 0.36 * (Income - 115000)
• 250000 <= Income → Tax = 79772 + 0.396 * (Income - 250000)

Variables, Types, Ranges

Dependency between variables

• Income is used to calculate Tax

Equivalence partitioning / Boundary analysis

Strategy

Test the boundaries (removing duplicates) → 10 tests

Test cases

Question

• Would you consider 22100 and 22099 a duplicate?

Another approach

One may argue that this function is continuous in all of its boundaries, which makes the result the same for all (on, off) pairs.

As, a consequence when looking just at the results at the boundaries we are not exercising them in the most efficient way. The whole purpose of boundary testing is to minimize the costs while maximizing fault detection capability.

What can we do to make our test cases more efficient? We can observe that what is really changing when making the transition from one partition to another is not the result of the function itself, but its derivative. Thus, we can focus on looking at the derivative for on and off points.

Let's recall the definition of the derivative of function f at point a: f'(a) = lim h -> 0 ((f(a+h) - f(a)) / h) To determine the derivative numerically we have to substitute h with sufficiently small number. For our case let h = 0.01.

We can now look at the boundary between first and second partition. off = 22099.99 on = 22100

By substituting those values into the definition of the derivative, we can determine the derivative of f at off on:

1. f'(off) = (f(off+h) - f(off)) / h = (f(22099.99 + 0.01) - f(22099.99)) / 0.01 = (f(22100) - f (22099.99)) / 0.01
2. f'(on) = (f(on+h) - f(a)) / h = (f(22100 + 0.01) - f(22100)) / 0.01 = (f(22100.01) - f (22100)) / 0.01

Now we find the expected derivatives for the corresponding off and on points from the specification:

1. f(off) = 0.15 * off thus f'(off) = 0.15
2. f(on) = 3315 + 0.28 * (on - 22100) thus f'(a) = 0.28

Now in our tests we can check whether actual derivatives for on and off points are equal to the expectation. What is left to do is to repeat the whole process for all other boundaries.

Note that if we decide to exercise the boundaries in this way, we have to remember to write the tests for the results too, as correct derivative does not imply correct function. Maybe the developer, during implementation, instead of 3315 + 0.28 * (on - 22100) wrote 315 + 0.28 * (on - 22100), which by testing only for derivatives we will not spot.

This example shows that as a software testers we, not only have to identify the boundaries between partitions, but also think about what is really changing when crossing those boundaries, in order to maximize the efficiency of our test cases.

Exercise 8: The ATM

An ATM allows you to withdraw 20 to 200 euros (inclusive) in increments of 20. For the example purposes, the program returns true or false, depending whether the amount required is valid.

Variables

Dependency among variables

Valid depends on the Amount

Equivalence Partitioning/Boundary Analysis

Note that -1 may be impossible to test.

Strategy

• Black-box testing: maybe all the on-points, off-points and invalid points.
• White-box testing: maybe just on-points, few-off points, invalid points.
  Why? If you know the implementation, you know how hard you need to test it.

Exercise 9: Piecewise

The input domain of a function is a set of all points (x, y) that meet the criteria:

• 1 < x <= 10
• 1 <= y <= 10
• y <= 14 - x

Variables

Dependency among variables

• x and y are dependent, since the range in y varies according to x.

Equivalence Partitioning/Boundary Analysis

Strategy

Make tests for all 12 boundaries

Another approach

We might look at the plot of the function. In the plot, we identify 5 boundaries (one at each of the extremes of the figure). As a tester, we can exercise these boundaries.

Exercise 10: Chocolate bars

A package should store a total number of kilograms. There are small bars (1 kg each) and big bars (5 kg each).

• The input of the program is the number of small bars and big bars available and the total number of kilos to store.
• We should calculate the number of small bars to use, assuming we always use big bars before small bars. Output -1, if it can't be done.

Dependency between variables

• Input variables are independent (they don't affect each other's range).
• Output variable depends on the three input values.
• Constraint: Use big bars before small bars.

Analyse it from the perspective of the output variable: how can the input variables influence the result?

Equivalence partitioning / Boundary analysis

Strategy

• There are 4 equivalence classes and 10 boundaries, but many of these boundaries are actually the same.
  • For example, boundary 1 and 5 are in fact the same boundary. Boundaries are not directional

Exercise 11: Median

A method takes a sorted int array as its only parameter and returns the median of the integers.

(You may assume that the parameter array is non-null and sorted)

For a sorted array of integers S the median is calculated as follows:

• if the length of S is odd, the median is the middle element of S.
• if the length of S is even, the median is computed as the average of the 2 middle integers of S.

A possible implementation could look like this:

/**
* Finds the median of a sorted int array.
*
* If the length of the array is odd, the method returns the element in the middle of the array as the median.
* If the length of the array is even, the method returns the average of the two middle elements of the array.
*
* @param array - The integer array to find the median from
* @return int - The median of the array
*/
public static int getMedian(int[] array) {
    // compute the median
    int n = array.length;
    if (n % 2 != 0) 
        return array[n / 2];
    return (array[n / 2 - 1] + array[n / 2]) / 2;
}

It becomes clear (by intuition) that the length of the array is of significant importance to the outcome. That's why I choose the length of the array as my variable.

Dependencies among variables

There is only 1 variable (1 parameter) so no other variable is present to even possibly influence this variable. --> No dependenies

Equivalence partitioning + boundary analysis

The boundary 'strictly positive even numbers' is a boundary with an infinite set of on- & off-points (think about the modulo operator!)

Strategy

• There are no dependencies to take into account.
• array.length has 4 boundaries to explore/test.
• 4 is not a lot (low time consumption) so I decide to test the 2 boundaries concerning the 'even number partition' again with bigger values. (This decision is tightly dependent on your time budget & test design)
• Total of 6 tests

Automated unit tests

/**
* Generates test cases for {@link #getMedian(String, int[], int)}.
*
* @return Stream of test cases
*/
private static Stream<Arguments> generator() {
    var t1 = Arguments.of("array length 1", new int[]{100}, 100);
    var t2 = Arguments.of("array length 2", new int[]{20, 100}, 60);
    var t3 = Arguments.of("array length 3", new int[]{1, 1, 3}, 1);
    var t4 = Arguments.of("array length 10", new int[]{2, 4, 6, 6, 6, 10, 10, 16, 20, 20}, 8);
    var t5 = Arguments.of("array length 11", new int[]{1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21}, 11);

    return Stream.of(t1, t2, t3, t4, t5);
}

/**
  * Parameterized test that uses {@link #generator()} to generate test cases.
  *
  * @param description Short description of the test case
  * @param array       Array in which to find the index of value
  * @param median      The expected median of array
  */
@ParameterizedTest(name = "{0}")
@MethodSource("generator")
void getMedian(String description, int[] array, int median) {
    assertThat(Training.getMedian(array)).isEqualTo(median);
}

/**
  * Checks if the IndexOutOfBounds exception is thrown when an empty array is provided.
  */
@Test
void arrayLengthZero() {
    assertThatExceptionOfType(IndexOutOfBoundsException.class).isThrownBy(() -> {
        int median = Training.getMedian(new int[]{});
    });
}

References

• Kaner, Cem, Sowmya Padmanabhan, and Douglas Hoffman. The Domain Testing Workbook. Context Driven Press, 2013.

• Kaner, Cem. What Is a Good Test Case?, 2003. URL: http://testingeducation.org/BBST/testdesign/Kaner_GoodTestCase.pdf