Property-Based Testing

Property-based testing checks rules that should hold across many inputs. Rather than write only “this input should produce this answer,” you describe a property and generate values with which to challenge it.

An image editor’s brightness control provides a small example. Its clamp operation keeps a value inside an inclusive interval: values below the minimum become the minimum, values above the maximum become the maximum, and values already inside stay unchanged.

Turn the requirement into several properties

For a valid interval, the answer must be inside the bounds. Clamping an answer again must not change it. An input already inside the interval must be returned unchanged.

These rules complement one another. A broken function that always returns the minimum would satisfy the first two. The third property prevents that particular mistake. Writing several weak properties does not automatically make a strong specification.

public static class Limits
{
    public static int Clamp(int value, int minimum, int maximum)
    {
        if (minimum > maximum) throw new ArgumentException("Reversed bounds.");
        if (value < minimum) return minimum;
        if (value > maximum) return maximum;
        return value;
    }
}

public static class Properties
{
    public static void CheckGeneratedInputs()
    {
        var random = new Random(742);
        for (int trial = 0; trial < 500; trial++)
        {
            int minimum = random.Next(-20, 21);
            int maximum = minimum + random.Next(0, 21);
            int value = random.Next(-50, 51);
            int result = Limits.Clamp(value, minimum, maximum);

            if (result < minimum || result > maximum)
                throw new Exception($"Outside bounds: {value}, {minimum}, {maximum}");
            if (Limits.Clamp(result, minimum, maximum) != result)
                throw new Exception("Clamping twice changed the result.");
            if (value >= minimum && value <= maximum && result != value)
                throw new Exception("An in-range input changed.");
        }
    }
}

Properties.CheckGeneratedInputs();

import random


def clamp(value, minimum, maximum):
    if minimum > maximum:
        raise ValueError("Reversed bounds.")
    if value < minimum:
        return minimum
    if value > maximum:
        return maximum
    return value


def check_generated_inputs():
    generator = random.Random(742)
    for _ in range(500):
        minimum = generator.randint(-20, 20)
        maximum = minimum + generator.randint(0, 20)
        value = generator.randint(-50, 50)
        result = clamp(value, minimum, maximum)

        assert minimum <= result <= maximum, (value, minimum, maximum)
        assert clamp(result, minimum, maximum) == result
        if minimum <= value <= maximum:
            assert result == value

check_generated_inputs()

A framework does more than this loop

The example exposes the mechanism without adding a library dependency. A property-testing framework such as Hypothesis also guides generation, remembers useful failures, and simplifies failing inputs. That last operation, often called shrinking, turns a complicated counterexample into a smaller one that still fails. 1

Our loop has no shrinker. It samples a small integer range and checks selected properties. Five hundred passes are not proof for every integer, overflow boundary, or invalid input.

Keep ordinary examples too: clamp(12, 0, 10) should return ten, and clamp(4, 0, 10) should return four. Examples communicate particular expectations; properties search more broadly for violations of relationships.

Find a property that is independent of the implementation

Useful candidates include encode/decode round trips, preserving a sum while moving values between accounts, or agreement between an optimized algorithm and a simpler reference algorithm. Be careful: two inverse functions can share a compensating mistake, and a reference algorithm can also be wrong.

Choose generators that include the shapes your code actually sees: empty collections, repeated values, unusual text, and boundary sizes. A thousand lists of five positive integers do not explore empty lists or negative values at all.

When a generated input exposes a defect, keep a small regression example alongside the broader property. The failure becomes both a discovered counterexample and a readable part of the specification.

Sources

1. Hypothesis — Introduction, on generated examples, properties, and simplified counterexamples.