diff options
Diffstat (limited to 'docs/source/quality.rst')
| -rw-r--r-- | docs/source/quality.rst | 200 |
1 files changed, 200 insertions, 0 deletions
diff --git a/docs/source/quality.rst b/docs/source/quality.rst new file mode 100644 index 0000000..328550c --- /dev/null +++ b/docs/source/quality.rst @@ -0,0 +1,200 @@ +.. _tools: + +******************************************** +Testing and Ensuring Type Annotation Quality +******************************************** + +Testing Annotation Accuracy +=========================== + +When creating a package with type annotations, authors may want to validate +that the annotations they publish meet their expectations. +This is especially important for library authors, for whom the published +annotations are part of the public interface to their package. + +There are several approaches to this problem, and this document will show +a few of them. + +.. note:: + + For simplicity, we will assume that type-checking is done with ``mypy``. + Many of these strategies can be applied to other type-checkers as well. + +Testing Using ``mypy --warn-unused-ignores`` +-------------------------------------------- + +Clever use of ``--warn-unused-ignores`` can be used to check that certain +expressions are or are not well-typed. + +The idea is to write normal python files which contain valid expressions along +with invalid expressions annotated with ``type: ignore`` comments. When +``mypy --warn-unused-ignores`` is run on these files, it should pass. +A directory of test files, ``typing_tests/``, can be maintained. + +This strategy does not offer strong guarantees about the types under test, but +it requires no additional tooling. + +If the following file is under test + +.. code-block:: python + + # foo.py + def bar(x: int) -> str: + return str(x) + +Then the following file tests ``foo.py``: + +.. code-block:: python + + bar(42) + bar("42") # type: ignore [arg-type] + bar(y=42) # type: ignore [call-arg] + r1: str = bar(42) + r2: int = bar(42) # type: ignore [assignment] + +Checking ``reveal_type`` output from ``mypy.api.run`` +----------------------------------------------------- + +``mypy`` provides a subpackage named ``api`` for invoking ``mypy`` from a +python process. In combination with ``reveal_type``, this can be used to write +a function which gets the ``reveal_type`` output from an expression. Once +that's obtained, tests can assert strings and regular expression matches +against it. + +This approach requires writing a set of helpers to provide a good testing +experience, and it runs mypy once per test case (which can be slow). +However, it builds only on ``mypy`` and the test framework of your choice. + +The following example could be integrated into a testsuite written in +any framework: + +.. code-block:: python + + import re + from mypy import api + + def get_reveal_type_output(filename): + result = api.run([filename]) + stdout = result[0] + match = re.search(r'note: Revealed type is "([^"]+)"', stdout) + assert match is not None + return match.group(1) + + +For example, we can use the above to provide a ``run_reveal_type`` pytest +fixture which generates a temporary file and uses it as the input to +``get_reveal_type_output``: + +.. code-block:: python + + import os + import pytest + + @pytest.fixture + def _in_tmp_path(tmp_path): + cur = os.getcwd() + try: + os.chdir(tmp_path) + yield + finally: + os.chdir(cur) + + @pytest.fixture + def run_reveal_type(tmp_path, _in_tmp_path): + content_path = tmp_path / "reveal_type_test.py" + + def func(code_snippet, *, preamble = ""): + content_path.write_text(preamble + f"reveal_type({code_snippet})") + return get_reveal_type_output("reveal_type_test.py") + + return func + + +For more details, see `the documentation on mypy.api +<https://mypy.readthedocs.io/en/stable/extending_mypy.html#integrating-mypy-into-another-python-application>`_. + +pytest-mypy-plugins +------------------- + +`pytest-mypy-plugins <https://github.com/typeddjango/pytest-mypy-plugins>`_ is +a plugin for ``pytest`` which defines typing test cases as YAML data. +The test cases are run through ``mypy`` and the output of ``reveal_type`` can +be asserted. + +This project supports complex typing arrangements like ``pytest`` parametrized +tests and per-test ``mypy`` configuration. It requires that you are using +``pytest`` to run your tests, and runs ``mypy`` in a subprocess per test case. + +This is an example of a parametrized test with ``pytest-mypy-plugins``: + +.. code-block:: yaml + + - case: with_params + parametrized: + - val: 1 + rt: builtins.int + - val: 1.0 + rt: builtins.float + main: | + reveal_type({[ val }}) # N: Revealed type is '{{ rt }}' + +Improving Type Completeness +=========================== + +One of the goals of many libraries is to ensure that they are "fully type +annotated", meaning that they provide complete and accurate type annotations +for all functions, classes, and objects. Having full annotations is referred to +as "type completeness" or "type coverage". + +Here are some tips for increasing the type completeness score for your +library: + +- Make type completeness an output of your testing process. Several type + checkers have options for generating useful output, warnings, or even + reports. +- If your package includes tests or sample code, consider removing them + from the distribution. If there is good reason to include them, + consider placing them in a directory that begins with an underscore + so they are not considered part of your library’s interface. +- If your package includes submodules that are meant to be + implementation details, rename those files to begin with an + underscore. +- If a symbol is not intended to be part of the library’s interface and + is considered an implementation detail, rename it such that it begins + with an underscore. It will then be considered private and excluded + from the type completeness check. +- If your package exposes types from other libraries, work with the + maintainers of these other libraries to achieve type completeness. + +.. warning:: + + The ways in which different type checkers evaluate and help you achieve + better type coverage may differ. Some of the above recommendations may or + may not be helpful to you, depending on which type checking tools you use. + +``mypy`` disallow options +------------------------- + +``mypy`` offers several options which can detect untyped code. +More details can be found in `the mypy documentation on these options +<https://mypy.readthedocs.io/en/latest/command_line.html#untyped-definitions-and-calls>`_. + +Some basic usages which make ``mypy`` error on untyped data are:: + + mypy --disallow-untyped-defs + mypy --disallow-incomplete-defs + +``pyright`` type verification +----------------------------- + +pyright has a special command line flag, ``--verifytypes``, for verifying +type completeness. You can learn more about it from +`the pyright documentation on verifying type completeness +<https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#verifying-type-completeness>`_. + +``mypy`` reports +---------------- + +``mypy`` offers several options options for generating reports on its analysis. +See `the mypy documentation on report generation +<https://mypy.readthedocs.io/en/stable/command_line.html#report-generation>`_ for details. |