Test Your Python Code Using Hypothesis

Introduction¶

Testing is important, no matter which programming language or framework you use. But writing tests is not as easy as it sounds. By this, I do not refer to writing a single test in general, but the difficulty of testing all possible edge cases. Testing the obvious cases is merely a matter of diligence, but writing tests in a way that helps you discover bugs isn't that easy.

However, property-based testing may help you to write (simple) tests, which also cover the majority of edge cases and can indeed help you find bugs in your code.

The article at hand gives you a quick overview of property-based testing in general and provides a few examples showing how to approach it in Python using Hypothesis. However, the article does not provide you an in-depth view of property-based testing in Python or the Hypothesis library. It rather aims to be a starting point for your journey towards property-based testing.

Note: The source code used in the article can be found on GitHub.

How Do We Distinguish Tests?¶

Before we can dive into property-based testing, we need to know how tests are distinguished in general. There are different ways to group tests. The two most common ways are based on the testing approach and the level of testing. Let's start with the testing levels as most people already heard of them. In essence, four testing levels exist (although people might know or define other levels, too):

• unit testing
• integration testing
• system testing
• end-to-end testing

The idea behind different testing levels is to focus on different things. Unit testing focuses on a certain part or functionality of software. This can be a single function or parts of a function.

In contrast, integration testing focuses on the cooperation of software components via their interfaces. System testing goes even further and tests the system as a whole.

Now, we will have a look at the wide variety of testing approaches that exist.

The most common and known ones are static and dynamic testing. Static testing refers to reviews, proofreading, and code checkers like linters. If software or parts of it are actually executed, we call it dynamic testing. Consequently, writing unit and integration tests ranks among dynamic testing.

Another common approach is the box approach. Basically, it can be divided into white-box and black-box testing (and grey-box testing as a hybrid of both). White-box testing verifies the internal structure or working of a program. It is the opposite of black-box testing where the application is seen as a black-box and the interaction with it is tested. This means that functionalities are tested without knowledge of the internal implementation.

What Is Property-Based Testing?¶

Now, that we had a quick look at how to differentiate tests, you might ask: Okay, okay, but what is property-based testing?

Property-based testing is a bit different from other testing approaches. If you have ever written tests, you might know that you usually work with sample data with explicit values. What I mean by this is that you say something like if I call the function func() with the number two as its argument, then the result should be eight. This holds especially true if you focus on unit testing. However, property-based testing goes in a different direction.

Instead of testing a unit, component or system based on predefined data samples or input data, you test against types.

Using Hypothesis for Property-Based Testing¶

Let's take a simple example. Assume you have two functions increment() and decrement(). A sample implementation might look like this:

# increment_decrement.py

def increment(number: int) -> int:
    return number + 1


def decrement(number: int) -> int:
    return number - 1

You might write unit tests for both like the following:

# test_increment_decrement_pytest.py

from increment_decrement import decrement
from increment_decrement import increment

def test_increment():
    x = 5
    expected = 6
    actual = increment(x)
    assert actual == expected


def test_decrement():
    x = 5
    expected = 4
    actual = decrement(x)
    assert actual == expected

Note: The tests are written using the pytest framework [²].

Of course, you can write more tests to test both functions with different values or even parametrize your tests. However, in the end, you test both functions using predefined values.

Writing tests using a property-based testing library like Hypothesis is different. Here, you specify the types you are testing against and the way the software should work or behave. The library then generates random values in accordance with the specified types to actually test the functions. Thereby, you won't miss edge cases as they are tested once in a while.

Let's have a look at how to test our two functions using Hypothesis.

# test_increment_decrement_hypothesis.py

from hypothesis import given
import hypothesis.strategies as st

from increment_decrement import decrement
from increment_decrement import increment


@given(st.integers())
def test_increment(x):
    expected = x + 1
    actual = increment(x)
    assert actual == expected


@given(st.integers())
def test_decrement(x):
    expected = x - 1
    actual = decrement(x)
    assert actual == expected

As you can see, both tests have a parameter x. The value for x is generated by Hypothesis using its integers() strategy. Hypothesis provides different kinds of strategies. In essence, these strategies correspond to built-in types or other structures and generate random data matching the given type.

Sounds good, isn't it? However, what if we want to test a function with a specific value to ensure, it is working with that value, too? Hypothesis provides an @example() decorator, where you can define a value, which is passed to the corresponding function even if it is not part of the randomly generated test data set.

Let's take a quick example:

# div.py

def div(dividend: int, divisor: int) -> int:
    return dividend // divisor

We defined a function div(), which takes a dividend and a divisor and returns the quotient of both. Notice, that both parameters are integers and as the result should be an integer, too, we perform an integer division using Python's // operator.

In order to test the div() function, we create a new test file test_div.py and write a test called test_div().

# test_div.py

from hypothesis import example
from hypothesis import given
import hypothesis.strategies as st

from div import div


@given(dividend=st.integers(), divisor=st.integers())
def test_div(dividend, divisor):
    if divisor == 0:
        expected = -1
    else:
        expected = dividend // divisor
    actual = div(dividend, divisor)
    assert actual == expected

Again, we use Hypothesis' integers() strategy to generate the values for dividend and divisor. The test we wrote may or may not pass depending on the values generated by Hypothesis at execution time. To ensure that the value 0 is always passed to the div() function, we add @example(1, 0) to the test_div() function. Consequently, div() is at least once called with the value 0 for divisor even if it is not in the randomly generated data set.

If we run the test suite as it is, test_div() will always fail. So let's modify the div() function to handle this case and make the tests pass:

# div.py

def div(dividend: int, divisor: int) -> int:
    if divisor == 0:
        return -1
    return dividend // divisor

Summary¶

Congratulations, you have made it through the article! While reading the article you learnt what property-based testing is and why it is useful. Furthermore, you took a quick glance at the library Hypothesis, which allows you to write property-based tests and execute them alongside your pytest tests.

I hope you enjoyed reading the article. Feel free to share it with your friends and colleagues! Do you have feedback? I am eager to hear it! You can contact me via the contact form or other resources listed in the contact section.

If you have not already, consider following me on Twitter, where I am @DahlitzF or subscribing to my newsletter! Stay curious and keep coding!

References¶