Py.test Basics
Posted on Fri 27 April 2018 in Posts
So I'm fortunate enough to work for an employer who grants me access to Safari Books Online. The other day I was browsing the site to discover that they now have not only just books, but also online courses/webinars on a variety of topics.
I stumbled across this one on Py.test which is a tool that is increasingly gaining traction in the world of Python unit-testing, a world which I'm very much interested in. As such, took the plunge and did the webinar, and thought I'd recap some of the neat Py.test things that were new to me. Disclaimer: with the exception of a brief intro a colleague gave me at a previous job, I've had no experience with Py.test so this is all going to be pretty basic stuff.
Running tests
Initially I first heard of Py.test as an alternative test runner. The built
in unittest
module can be used for running your unit tests, but many people
(myself included) will use nose
as a tool for running their tests. For example, you might do something like
the following to run your tests:
$ nosetests
..................................
----------------------------------------------------------------------
Ran 34 tests in 1.440s
OK
With Py.test you'd do something like (from within the project's directory):
$ PYTHONPATH=. pytest
========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/block, inifile:
collected 3 items
test_block/test_block.py ... [100%]
======================================================================== 3 passed in 0.02 seconds ========================================================================
The output's quite different, but not in a bad way.
Tests as Functions
The first thing that'll jump out at you about Py.test is that it's encouraged
to write your tests as functions, rather than methods in a class. In the base
unittest
module in Python it follows the typical xUnit style pattern of test
organization: you create a class which inherits from some base class
(unittest.TestCase
in Python's unittest
module), and then each individual
unit test is a method on that class. So if you had a class Dog
you might
write a test class like the following:
class TestDog(unittest.TestCase):
def test_speak(self):
dog = Dog()
result = dog.speak()
self.assertEqual("bark!", result)
Certainly you can still write tests like this with Py.test (ie all old tests you have will still work just fine with Py.test), but instead you're encouraged to write your tests as standalone functions. So the above might look like:
def test_dog_speak():
dog = Dog()
result = dog.speak()
assert "bark!" == result
Note as well another key difference: the use of Python's built-in assert
statement rather than using the assert*()
methods defined in
unittest.TestCase
since we're no longer inheriting from that class.
Sidebar: The assert Gotcha
The assert statement gotcha: in Python the assert statement is a statement, not a function, so this can trip people up when they pass a second argument to it. For example:
assert somecondition, "This gets printed out"
will print "This gets printed out" if somecondition is false
, but
this is not the same as:
assert(somecondition, "This gets printed out")
which always evaluates to true because it's treated as passing a 2-value tuple to the assert statement, and a non-empty tuple is truthy in Python. This is a classic Python gotcha. Py.test adds some smarts around this, if we had a test like:
def test_assert_with_tuple():
assert (False == True, "Should be false")
Then we'd get output like:
========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/Integer, inifile:
collected 15 items
test/test_integr.py ............. [100%]
============================================================================ warnings summary ============================================================================
test/test_integr.py:119
assertion is always true, perhaps remove parentheses?
-- Docs: http://doc.pytest.org/en/latest/warnings.html
================================================================== 12 passed, 1 warnings in 0.11 seconds =================================================================
Note how Py.test warns us that we might be doing something dumb.
Testing Raised Exceptions
A common thing to do in unit tests is test if a particular exception is
raised under certain circumstances. You can certainly do this in vanilla
unittest
:
class TestFoo(unittest.TestCase):
def test_foo_throws_value_error_when_given_negative_number(self):
# test will only pass if foo() raises ValueError
with self.assertRaises(ValueError):
foo(-1)
But again since we're no longer inheriting from unittest.TestCase
how do
we do this with Py.test? There's a couple ways, one is still with a context
manager:
import pytest
def test_foo_throws_value_error_when_given_negative_number():
# test will only pass if foo() raises ValueError
with pytest.raises(ValueError):
foo(-1)
Another is to "mark" the test as an expected failure:
@pytest.mark.xfail(raises=ValueError)
def test_foo_throws_value_error_when_given_negative_number():
foo(-1)
The latter is slighly different, and this is reported in the test results as you'll see output like:
========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/Integer, inifile:
collected 15 items
test/test_integr.py ..x............ [100%]
============================================================ 12 passed, 1 skipped, 1 xfailed in 0.12 seconds =============================================================
Note that 1 xfailed
bit, indicating that there was a single test with an "expected"
failure. This is effectively coding a way in which the SUT fails, but in an expected
way.
The docs
suggest that:
Using pytest.raises is likely to be better for cases where you are testing exceptions your own code is deliberately raising, whereas using @pytest.mark.xfail with a check function is probably better for something like documenting unfixed bugs (where the test describes what “should” happen) or bugs in dependencies.
I could see this being useful for a test that's currently failing in a way you
expect, you could comment the test out (but then you might as well delete it) or
you could codify that you expect it to fail. The raises:ValueError)
restricts
the way you expect it to fail (for example if you expect it to throw a ValueError,
and it throws a RuntimeError then that's not expected and should cause a test run
failure).
Forcing a Failure
You can also force a test to fail:
def test_force_failure():
# This test will always fail
pytest.fail("this will fail")
This is effectively the same thing as adding self.assertEqual(True, False)
in a vanilla Python unit test. I'm not sure where this would be useful, the instructor
gave this example:
def test_import_error():
try:
import somethirdpartylibrary
except ImportError:
pytest.fail("No module named somethirdpartylibrary to import")
But I fail to see the benefit of this. If the import was missing, you'll still get
test failures (in fact, presumably any test that needs somethirdpartylibrary
will
still blow up even with this test in place).
Approximations
Sometimes you want to test a floating point number for a particular value in a test,
but because floating point values in any programming language are not precise, you
effectively have to create a "tolerable range" to test against. For example in plain
Python unitest
you might do something like:
self.assertAlmostEqual(some_value, some_other_value, delta=0.001)
which would pass if some_value
and some_other_value
are within +/- 0.001
of
each other. With Py.test you can do the following:
def test_approx_same():
expected = 0.3
result = 0.1 + 0.2
assert pytest.approx(expected) == result
There's a bunch of flexibility around how to use approx
and it works with more
than just floating point numbers.
The docs are quite good.
Marking Tests & Selective Test Execution
Oftentimes you want to group tests into various categories. For example you might want
to separate unit tests from integration tests, or fast-running tests from slow-running
tests, etc. Doing this in vanila unittest
is kinda cumbersome, but with Py.test it's
easy:
@pytest.mark.long
def test_this_takes_long():
print("this is taking a long time....")
import time ; time.sleep(10)
assert True == True
@pytest.mark.fast
def test_this_is_fast():
assert True == True
With tests like this you can run just the fast tests:
pytest -m fast
or just the slow tests:
pytest -m slow
Or even do some simple boolean logic, like for example anything that's not in the slow category:
pytest -m "not slow"
Really handy stuff.
You can also do fuzzy matching of tests by name, for example to run all the tests with the word "bad" in them:
pytest -k bad
This will collect & run tests with the word bad
in the name (ex: test_bad()
,
test_that_the_bad_stuff_doesnt_happen()
, etc). Really useful for when you just want
to quickly run one or two specific tests.
Conditionally Skipping Tests
You can also conditionally skip tests:
@pytest.mark.skipif(
sys.version_info < (3,6),
reason="Requires Python 3.6"
)
def test_that_requires_python_3_6_because_f_strings():
result = function_that_uses_f_strings()
assert "some expected value" == result
Since the block in the skipif
is arbitrary Python code, you could use this for something
like running tests only if a particular environment variable is set, etc.
Parameterized Tests
This is where we really start to see some cool stuff. If you've ever written unit tests
in jUnit you'll have probably at some point come across
parameterized tests
which is a really useful technique for reducing test boilerplate by separating test
definition from test input data. Let's say we had a method add()
that adds two numbers
together & returns the result (I know it's a boring example, but it really illustrates
the technique). You might write two tests for this function to test two positive numbers
and two negative numbers:
class TestAdd(unittest.TestCase):
def test_add_positive(self):
expected = 4
result = add(1, 3)
self.assertEqual(expected, result)
def test_add_negative(self):
expected = -4
result = add(-1, -3)
self.assertEqual(expected, result)
Note that these two tests are identical save the expected and input values. Imagine if you could somehow make those parameters to a test, then you might do something like:
def test_add(x, y, expected_result):
result = add(x, y)
assert expected_result == result
The question then becomes how do we specify those input values. In Python 3.4 they added
something like this with subTest
(docs):
def test_add(self):
values_to_test = [
(1, 3, 4),
(-1, -3, -4),
]
for test_vals in values_to_test:
with self.subTest(test_vals=test_vals):
x,y,expected_result = test_vals
result = add(x,y)
self.assertEqual(expected_result, result)
But this is super clunky, the test data is within the test, obfuscating the meaning
of the actual test. You also have to manually unpack everything, and when we run it,
it ends up looking like a single test instead of two (so if one fails it's harder to
figure out which one of the two tuples in values_to_test
caused the failure).
Instead, with Py.test you can use the parametrize
mark:
@pytest.mark.parametrize('x, y, expected_result', [
(1, 3, 4),
(-1, -3, -4),
])
def test_add(x, y, expected_result):
result = add(x, y)
assert expected_result == result
This is much cleaner, the test meaning is extremely clear, and if we use the -v
flag
when running the tests we can see that this produces two distinct tests for the two cases:
========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0 -- /Users/aparkin/.virtualenvs/pytestcourse/bin/python3.6
cachedir: .pytest_cache
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/Integer, inifile:
collected 2 items
test/test_integr.py::test_add[1-3-4] PASSED [ 50%]
test/test_integr.py::test_add[-1--3--4] PASSED [100%]
======================================================================== 2 passed in 0.02 seconds ========================================================================
Also note that the input itself for the test is in the test identifier (ie the [1-3-4]
for the test which added 1 & 3 to get 4, etc).
Now adding additional test cases is a simple matter of adding another tuple to the parametrize decorator. That is: each new test becomes a single line. That's some super-concise test definition.
The docs on this illustrate more stuff you can do with it, I'm just barely scratching the surface.
Debugging
Another handy trick is that with the --pdb
command-line argument you can actually have
Py.test automatically drop into the pdb debugger on a failing test:
$ pytest -v --pdb
========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0 -- /Users/aparkin/.virtualenvs/pytestcourse/bin/python3.6
cachedir: .pytest_cache
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/Integer, inifile:
collected 4 items
test/test_integr.py::test_add[1-3-4] PASSED [ 25%]
test/test_integr.py::test_add[-1--3--4] PASSED [ 50%]
test/test_integr.py::test_fail FAILED [ 75%]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
def test_fail():
x = 2342423
> assert 42 == 4234
E assert 42 == 4234
test/test_integr.py:167: AssertionError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/aparkin/temp/sandbox/pytestcourse/Integer/test/test_integr.py(167)test_fail()
-> assert 42 == 4234
(Pdb)
You can then use the usual pdb
commands to try and figure how why a test is failing.
Other Stuff
There's way more to Py.test, things like Fixtures to replace the setUp
& tearDown
for
classes, running doctests, and a million other options. Hopefully this is a good starting
introduction.