In the world of testing it is important to write tests that are both easy to read and covering a wide range of scenarios. Often one of these will be sacrificed to facilitate the other, such as hard coding your examples so that your test logic remains clear or by creating an overly complicated setup so that multiple scenarios can be explored. Here we discuss two tools that, when combined, will allow you to explore more of the test surface of your web API while still creating clear and maintainable tests.
Hypothesis is a library that, when provided with a description of the parameters for your API will explore many situations that will stress your system allowing you to explore your API more thoroughly. When you have errors, hypothesis will then work to simplify the failing example to show you the simplest failing case and then keep this failing case in it's database to use until the problem is resolved.
Hypothesis uses, what it calls strategies to inject parameters into your test cases. These strategies produce 200 different random examples values from across the variable space (including 'nasty' values such as min, max, nan and inf) and your test is run for each example. In general this is done using a single decorator,
@given, on your test case which passes the parameters into the test via the argument list. In the following example two float values are passed to
These can be passed in as either positional or named parameters (though it is not advised to mix) so the previous test can be defined as:
Hypothesis comes with strategies for all built in python types as well as mechanisms for creating complex data structures. It is simple to
chain strategies together to produce new strategies. For example, if we had an object that could be costructed with two keyword parameters
float_param we could create a strategy for it by chaining dictionary, text and float strategies.
This will take an example from the
fixed_dictionary strategy which has\
float_param fields and pass it into the
flatmap function. This will then be used to generate a new strategy that consists of our new instance of
MyObj. It is also possible to create entirely new strategies - a good description of this can be found
in the docs here. There is also built in support for generating django models in the
In general we want to test properties of our methods using hypothesis to avoid replicating the implementation in the test itself, this will help keep tests clear and independent of the implementation. For example we would probably not want to test:
While it would be ok to write this test as it is sufficiently simple, for more complex examples such as encoding and decoding data you would likely not want to test the encoded result, instead you would test that given a value, encoding and decoding gives the original result:
Testing like this gives us two major advantages:
There are many good articles on property driven testing so we will not go into it any further here. The full documentation for hypothesis can be found here.
It is also easy to hide away request construction in the test framework. By doing this it is difficult to know whether we are actually testing the API in a way it will be used by the consumers or whether we are just hiding in some ideal world. Remember, we need to be nasty to our applications in order to test them correctly. Of cource we can enforce all of this in our testing framework but often this makes the test a lot harder to read.
Gabbi is a tool for declaratively creating tests for web APIs. It hopes to solve these problems. To do this Gabbi uses yaml files to declare the test API calls and the expected response which can be run using
gabbi-run. For example, lets take a look at a simple web service which has a database of
Thing\ s. To test the creation of a thing we may have something like:
Here we are creating a
Thing, making sure the request is a
POST, the response code is 201 and that the correct content type is being sent. We also make sure that if we make a
GET (default) request for the
Thing created in the previous request by using
$RESPONSE, we get a 200 status code (default) with json data that has the correct
name. In this example
$.name is used to reference the
name property from the responses json data. This is json path lookup notation, for more information take a look at the
JSONPath RW docs pages <http://jsonpath-rw.readthedocs.org/en/latest/>__.
Full Gabbi documentation can be found here.
The rest of this post will explore how these two tools can be used to create readable, parameterised tests for http API methods. The example is for a django project using django rest framework for the web API and can be found at https://github.com/wildfish/gabbi-hypothesis-demo.
First thing we need to do is create a custom test case that will handle the hard work by creating the Gabbi test cases from a python dictionary rather than yaml files:
First thing to note, our test case inherits from the django
TestCase supplied by Hypothesis as well as the
LiveServerTestCase which will handle the database between Hypothesis runs as well as creating the
server to test against. Next we initialise the Gabbi request handlers. Third we build the Gabbi test suite from the declaration and finally run the tests reporting any errors so that Hypothesis can try and find a simpler error case.
If we go back to the example from above with a web app that stores
Thing\ s we would start by writing:
This is exactly the same test as above except we are taking the name for our object from Hypothesis. When we run the test we get the following output:
This gives us 2 pieces of information, firstly that it failed and secondly, that it failed because the API was passed an empty string. This has shown us that our assumptions about what data can be handled by our API were false. Now we can add a new test case to cover the empty string example and refine our original test to ignore the value. It turns out that django rest framework will strip input of white space on cleaning the data so we will modify our tests to account for all 'empty' strings.
To update our test we will introduce
assume from Hypothesis.
filter as you may expect, only gives values that match the filter function.
assume fails the test in a way that Hypothesis will catch and then learn to favour giving examples that match the assumption. This failure is not recorded as a failure and does not take up one of the test examples:
Now, in the first test case any empty strings will fail the assumption and be reported to Hypothesis. In the second test case we are pre filtering the examples so that only empty strings are passed in. When we run these tests we get:
So another assumption about our API was not correct. Our API cannot handle arbitrarily long names. Again we add a test to cover this example and refine the original test:
When we run this we now all tests pass.
Let's assume that this API was to be used by a wide audience and we wanted to change our model so that
name was actually stored as
last_name. We would be safe in making that change because our tests never use the model directly, the only thing we state in our tests is that using this version of the API our consumers will be able to create an object by supplying the
name parameter and when retrieving the object the
name parameter will be present with the correct value. If all tests pass we should be able to roll out our model change safe in the knowledge our users using the old API will not have their applications immediately break.
We have seen how to combine Hypothesis and Gabbi to create tests for our API that are both readable and have high coverage by delegating the value selection. We have also seen how using these methods we can check and improve the assumptions we have made about our API. Finally, property based tesing can be used to help keep test faith for older versions of you API.
Using these tools along with test driven development can ensure we get the API that we want to use and have great test coverage.
It turns out some of our hypothesis examples can be made simpler. Our chaining example can be created using the
We can also make the 'empty' string case simpler by specifying an
alphabet rather than using