How do you generate dynamic (parameterized) unit tests in Python?

How do you generate dynamic (parameterized) unit tests in Python?

This is called parametrization.

There are several tools that support this approach. E.g.:

• pytests decorator
• parameterized

The resulting code looks like this:

from parameterized import parameterized

class TestSequence(unittest.TestCase):
    @parameterized.expand([
        [foo, a, a,],
        [bar, a, b],
        [lee, b, b],
    ])
    def test_sequence(self, name, a, b):
        self.assertEqual(a,b)

Which will generate the tests:

test_sequence_0_foo (__main__.TestSequence) ... ok
test_sequence_1_bar (__main__.TestSequence) ... FAIL
test_sequence_2_lee (__main__.TestSequence) ... ok

======================================================================
FAIL: test_sequence_1_bar (__main__.TestSequence)
----------------------------------------------------------------------
Traceback (most recent call last):
  File /usr/local/lib/python2.7/site-packages/parameterized/parameterized.py, line 233, in <lambda>
    standalone_func = lambda *a: func(*(a + p.args), **p.kwargs)
  File x.py, line 12, in test_sequence
    self.assertEqual(a,b)
AssertionError: a != b

For historical reasons Ill leave the original answer circa 2008):

I use something like this:

import unittest

l = [[foo, a, a,], [bar, a, b], [lee, b, b]]

class TestSequense(unittest.TestCase):
    pass

def test_generator(a, b):
    def test(self):
        self.assertEqual(a,b)
    return test

if __name__ == __main__:
    for t in l:
        test_name = test_%s % t[0]
        test = test_generator(t[1], t[2])
        setattr(TestSequense, test_name, test)
    unittest.main()

Using unittest (since 3.4)

Since Python 3.4, the standard library unittest package has the subTest context manager.

See the documentation:

• 26.4.7. Distinguishing test iterations using subtests
• subTest

Example:

from unittest import TestCase

param_list = [(a, a), (a, b), (b, b)]

class TestDemonstrateSubtest(TestCase):
    def test_works_as_expected(self):
        for p1, p2 in param_list:
            with self.subTest():
                self.assertEqual(p1, p2)

You can also specify a custom message and parameter values to subTest():

with self.subTest(msg=Checking if p1 equals p2, p1=p1, p2=p2):

Using nose

The nose testing framework supports this.

Example (the code below is the entire contents of the file containing the test):

param_list = [(a, a), (a, b), (b, b)]

def test_generator():
    for params in param_list:
        yield check_em, params[0], params[1]

def check_em(a, b):
    assert a == b

The output of the nosetests command:

> nosetests -v
testgen.test_generator(a, a) ... ok
testgen.test_generator(a, b) ... FAIL
testgen.test_generator(b, b) ... ok

======================================================================
FAIL: testgen.test_generator(a, b)
----------------------------------------------------------------------
Traceback (most recent call last):
  File /usr/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py, line 203, in runTest
    self.test(*self.arg)
  File testgen.py, line 7, in check_em
    assert a == b
AssertionError

----------------------------------------------------------------------
Ran 3 tests in 0.006s

FAILED (failures=1)

How do you generate dynamic (parameterized) unit tests in Python?

This can be solved elegantly using Metaclasses:

import unittest

l = [[foo, a, a,], [bar, a, b], [lee, b, b]]

class TestSequenceMeta(type):
    def __new__(mcs, name, bases, dict):

        def gen_test(a, b):
            def test(self):
                self.assertEqual(a, b)
            return test

        for tname, a, b in l:
            test_name = test_%s % tname
            dict[test_name] = gen_test(a,b)
        return type.__new__(mcs, name, bases, dict)

class TestSequence(unittest.TestCase):
    __metaclass__ = TestSequenceMeta

if __name__ == __main__:
    unittest.main()