1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
|
.. _`test generators`:
.. _`parametrizing-tests`:
.. _`parametrized test functions`:
.. _`parametrize`:
.. _`parametrize-basics`:
Parametrizing fixtures and test functions
==========================================================================
pytest enables test parametrization at several levels:
- :py:func:`pytest.fixture` allows one to :ref:`parametrize fixture
functions <fixture-parametrize>`.
* `@pytest.mark.parametrize`_ allows one to define multiple sets of
arguments and fixtures at the test function or class.
* `pytest_generate_tests`_ allows one to define custom parametrization
schemes or extensions.
.. _parametrizemark:
.. _`@pytest.mark.parametrize`:
``@pytest.mark.parametrize``: parametrizing test functions
---------------------------------------------------------------------
.. regendoc: wipe
.. versionadded:: 2.2
.. versionchanged:: 2.4
Several improvements.
The builtin ``pytest.mark.parametrize`` decorator enables
parametrization of arguments for a test function. Here is a typical example
of a test function that implements checking that a certain input leads
to an expected output::
# content of test_expectation.py
import pytest
@pytest.mark.parametrize("test_input,expected", [
("3+5", 8),
("2+4", 6),
("6*9", 42),
])
def test_eval(test_input, expected):
assert eval(test_input) == expected
Here, the ``@parametrize`` decorator defines three different ``(test_input,expected)``
tuples so that the ``test_eval`` function will run three times using
them in turn::
$ pytest
======= test session starts ========
platform linux -- Python 3.x.y, pytest-3.x.y, py-1.x.y, pluggy-0.x.y
rootdir: $REGENDOC_TMPDIR, inifile:
collected 3 items
test_expectation.py ..F
======= FAILURES ========
_______ test_eval[6*9-42] ________
test_input = '6*9', expected = 42
@pytest.mark.parametrize("test_input,expected", [
("3+5", 8),
("2+4", 6),
("6*9", 42),
])
def test_eval(test_input, expected):
> assert eval(test_input) == expected
E AssertionError: assert 54 == 42
E + where 54 = eval('6*9')
test_expectation.py:8: AssertionError
======= 1 failed, 2 passed in 0.12 seconds ========
As designed in this example, only one pair of input/output values fails
the simple test function. And as usual with test function arguments,
you can see the ``input`` and ``output`` values in the traceback.
Note that you could also use the parametrize marker on a class or a module
(see :ref:`mark`) which would invoke several functions with the argument sets.
It is also possible to mark individual test instances within parametrize,
for example with the builtin ``mark.xfail``::
# content of test_expectation.py
import pytest
@pytest.mark.parametrize("test_input,expected", [
("3+5", 8),
("2+4", 6),
pytest.param("6*9", 42,
marks=pytest.mark.xfail),
])
def test_eval(test_input, expected):
assert eval(test_input) == expected
Let's run this::
$ pytest
======= test session starts ========
platform linux -- Python 3.x.y, pytest-3.x.y, py-1.x.y, pluggy-0.x.y
rootdir: $REGENDOC_TMPDIR, inifile:
collected 3 items
test_expectation.py ..x
======= 2 passed, 1 xfailed in 0.12 seconds ========
The one parameter set which caused a failure previously now
shows up as an "xfailed (expected to fail)" test.
To get all combinations of multiple parametrized arguments you can stack
``parametrize`` decorators::
import pytest
@pytest.mark.parametrize("x", [0, 1])
@pytest.mark.parametrize("y", [2, 3])
def test_foo(x, y):
pass
This will run the test with the arguments set to ``x=0/y=2``, ``x=0/y=3``, ``x=1/y=2`` and
``x=1/y=3``.
.. _`pytest_generate_tests`:
Basic ``pytest_generate_tests`` example
---------------------------------------------
Sometimes you may want to implement your own parametrization scheme
or implement some dynamism for determining the parameters or scope
of a fixture. For this, you can use the ``pytest_generate_tests`` hook
which is called when collecting a test function. Through the passed in
``metafunc`` object you can inspect the requesting test context and, most
importantly, you can call ``metafunc.parametrize()`` to cause
parametrization.
For example, let's say we want to run a test taking string inputs which
we want to set via a new ``pytest`` command line option. Let's first write
a simple test accepting a ``stringinput`` fixture function argument::
# content of test_strings.py
def test_valid_string(stringinput):
assert stringinput.isalpha()
Now we add a ``conftest.py`` file containing the addition of a
command line option and the parametrization of our test function::
# content of conftest.py
def pytest_addoption(parser):
parser.addoption("--stringinput", action="append", default=[],
help="list of stringinputs to pass to test functions")
def pytest_generate_tests(metafunc):
if 'stringinput' in metafunc.fixturenames:
metafunc.parametrize("stringinput",
metafunc.config.getoption('stringinput'))
If we now pass two stringinput values, our test will run twice::
$ pytest -q --stringinput="hello" --stringinput="world" test_strings.py
..
2 passed in 0.12 seconds
Let's also run with a stringinput that will lead to a failing test::
$ pytest -q --stringinput="!" test_strings.py
F
======= FAILURES ========
_______ test_valid_string[!] ________
stringinput = '!'
def test_valid_string(stringinput):
> assert stringinput.isalpha()
E AssertionError: assert False
E + where False = <built-in method isalpha of str object at 0xdeadbeef>()
E + where <built-in method isalpha of str object at 0xdeadbeef> = '!'.isalpha
test_strings.py:3: AssertionError
1 failed in 0.12 seconds
As expected our test function fails.
If you don't specify a stringinput it will be skipped because
``metafunc.parametrize()`` will be called with an empty parameter
list::
$ pytest -q -rs test_strings.py
s
======= short test summary info ========
SKIP [1] test_strings.py:2: got empty parameter set ['stringinput'], function test_valid_string at $REGENDOC_TMPDIR/test_strings.py:1
1 skipped in 0.12 seconds
For further examples, you might want to look at :ref:`more
parametrization examples <paramexamples>`.
.. _`metafunc object`:
The **metafunc** object
-------------------------------------------
.. currentmodule:: _pytest.python
.. autoclass:: Metafunc
:members:
|
