1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
|
.. _tools:
********************************************
Testing and Ensuring Type Annotation Quality
********************************************
Testing Annotation Accuracy
===========================
When creating a package with type annotations, authors may want to validate
that the annotations they publish meet their expectations.
This is especially important for library authors, for whom the published
annotations are part of the public interface to their package.
There are several approaches to this problem, and this document will show
a few of them.
.. note::
For simplicity, we will assume that type-checking is done with ``mypy``.
Many of these strategies can be applied to other type-checkers as well.
Testing Using ``mypy --warn-unused-ignores``
--------------------------------------------
Clever use of ``--warn-unused-ignores`` can be used to check that certain
expressions are or are not well-typed.
The idea is to write normal python files which contain valid expressions along
with invalid expressions annotated with ``type: ignore`` comments. When
``mypy --warn-unused-ignores`` is run on these files, it should pass.
A directory of test files, ``typing_tests/``, can be maintained.
This strategy does not offer strong guarantees about the types under test, but
it requires no additional tooling.
If the following file is under test
.. code-block:: python
# foo.py
def bar(x: int) -> str:
return str(x)
Then the following file tests ``foo.py``:
.. code-block:: python
bar(42)
bar("42") # type: ignore [arg-type]
bar(y=42) # type: ignore [call-arg]
r1: str = bar(42)
r2: int = bar(42) # type: ignore [assignment]
Checking ``reveal_type`` output from ``mypy.api.run``
-----------------------------------------------------
``mypy`` provides a subpackage named ``api`` for invoking ``mypy`` from a
python process. In combination with ``reveal_type``, this can be used to write
a function which gets the ``reveal_type`` output from an expression. Once
that's obtained, tests can assert strings and regular expression matches
against it.
This approach requires writing a set of helpers to provide a good testing
experience, and it runs mypy once per test case (which can be slow).
However, it builds only on ``mypy`` and the test framework of your choice.
The following example could be integrated into a testsuite written in
any framework:
.. code-block:: python
import re
from mypy import api
def get_reveal_type_output(filename):
result = api.run([filename])
stdout = result[0]
match = re.search(r'note: Revealed type is "([^"]+)"', stdout)
assert match is not None
return match.group(1)
For example, we can use the above to provide a ``run_reveal_type`` pytest
fixture which generates a temporary file and uses it as the input to
``get_reveal_type_output``:
.. code-block:: python
import os
import pytest
@pytest.fixture
def _in_tmp_path(tmp_path):
cur = os.getcwd()
try:
os.chdir(tmp_path)
yield
finally:
os.chdir(cur)
@pytest.fixture
def run_reveal_type(tmp_path, _in_tmp_path):
content_path = tmp_path / "reveal_type_test.py"
def func(code_snippet, *, preamble = ""):
content_path.write_text(preamble + f"reveal_type({code_snippet})")
return get_reveal_type_output("reveal_type_test.py")
return func
For more details, see `the documentation on mypy.api
<https://mypy.readthedocs.io/en/stable/extending_mypy.html#integrating-mypy-into-another-python-application>`_.
pytest-mypy-plugins
-------------------
`pytest-mypy-plugins <https://github.com/typeddjango/pytest-mypy-plugins>`_ is
a plugin for ``pytest`` which defines typing test cases as YAML data.
The test cases are run through ``mypy`` and the output of ``reveal_type`` can
be asserted.
This project supports complex typing arrangements like ``pytest`` parametrized
tests and per-test ``mypy`` configuration. It requires that you are using
``pytest`` to run your tests, and runs ``mypy`` in a subprocess per test case.
This is an example of a parametrized test with ``pytest-mypy-plugins``:
.. code-block:: yaml
- case: with_params
parametrized:
- val: 1
rt: builtins.int
- val: 1.0
rt: builtins.float
main: |
reveal_type({[ val }}) # N: Revealed type is '{{ rt }}'
Improving Type Completeness
===========================
One of the goals of many libraries is to ensure that they are "fully type
annotated", meaning that they provide complete and accurate type annotations
for all functions, classes, and objects. Having full annotations is referred to
as "type completeness" or "type coverage".
Here are some tips for increasing the type completeness score for your
library:
- Make type completeness an output of your testing process. Several type
checkers have options for generating useful output, warnings, or even
reports.
- If your package includes tests or sample code, consider removing them
from the distribution. If there is good reason to include them,
consider placing them in a directory that begins with an underscore
so they are not considered part of your library’s interface.
- If your package includes submodules that are meant to be
implementation details, rename those files to begin with an
underscore.
- If a symbol is not intended to be part of the library’s interface and
is considered an implementation detail, rename it such that it begins
with an underscore. It will then be considered private and excluded
from the type completeness check.
- If your package exposes types from other libraries, work with the
maintainers of these other libraries to achieve type completeness.
.. warning::
The ways in which different type checkers evaluate and help you achieve
better type coverage may differ. Some of the above recommendations may or
may not be helpful to you, depending on which type checking tools you use.
``mypy`` disallow options
-------------------------
``mypy`` offers several options which can detect untyped code.
More details can be found in `the mypy documentation on these options
<https://mypy.readthedocs.io/en/latest/command_line.html#untyped-definitions-and-calls>`_.
Some basic usages which make ``mypy`` error on untyped data are::
mypy --disallow-untyped-defs
mypy --disallow-incomplete-defs
``pyright`` type verification
-----------------------------
pyright has a special command line flag, ``--verifytypes``, for verifying
type completeness. You can learn more about it from
`the pyright documentation on verifying type completeness
<https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#verifying-type-completeness>`_.
``mypy`` reports
----------------
``mypy`` offers several options options for generating reports on its analysis.
See `the mypy documentation on report generation
<https://mypy.readthedocs.io/en/stable/command_line.html#report-generation>`_ for details.
|
