diff options
Diffstat (limited to 'docs/source/libraries.rst')
| -rw-r--r-- | docs/source/libraries.rst | 602 |
1 files changed, 602 insertions, 0 deletions
diff --git a/docs/source/libraries.rst b/docs/source/libraries.rst new file mode 100644 index 0000000..96e5370 --- /dev/null +++ b/docs/source/libraries.rst @@ -0,0 +1,602 @@ +.. _libraries: + +*********************** +Typing Python Libraries +*********************** + +Much of Python’s popularity can be attributed to the rich collection of +Python libraries available to developers. Authors of these libraries +play an important role in improving the experience for Python +developers. This document provides some recommendations and guidance for +Python library authors. + +These recommendations are intended to provide the following benefits: + +1. Consumers of libraries should have a great coding experience with + fast and accurate completion suggestions, class and function + documentation, signature help (including parameter default values), + hover text, and auto-imports. This should happen by default without + needing to download extra packages and without any special + configuration. These features should be consistent across the Python + ecosystem regardless of a developer’s choice of editor, IDE, notebook + environment, etc. +2. Consumers of libraries should be able to rely on complete and + accurate type information so static type checkers can detect and + report type inconsistencies and other violations of the interface + contract. +3. Library authors should be able to specify a well-defined interface + contract that is enforced by tools. This allows a library + implementation to evolve and improve without breaking consumers of + the library. +4. Library authors should have the benefits of static type checking to + produce high-quality, bug-free implementations. + +Inlined Type Annotations and Type Stubs +======================================= + +`PEP 561 <https://www.python.org/dev/peps/pep-0561/>`__ documents +several ways type information can be delivered for a library: inlined +type annotations, type stub files included in the package, a separate +companion type stub package, and type stubs in the typeshed repository. +Some of these options fall short on delivering the benefits above. We +therefore provide the following more specific guidance to library +authors. + +.. note:: + All libraries should include inlined type annotations for the + functions, classes, methods, and constants that comprise the public + interface for the library. + +Inlined type annotations should be included directly within the source +code that ships with the package. Of the options listed in PEP 561, +inlined type annotations offer the most benefits. They typically require +the least effort to add and maintain, they are always consistent with +the implementation, and docstrings and default parameter values are +readily available, allowing language servers to enhance the development +experience. + +There are cases where inlined type annotations are not possible — most +notably when a library’s exposed functionality is implemented in a +language other than Python. + +.. note:: + Libraries that expose symbols implemented in languages other than + Python should include stub (``.pyi``) files that describe the types for + those symbols. These stubs should also contain docstrings and default + parameter values. + +In many existing type stubs (such as those found in typeshed), default +parameter values are replaced with with ``...`` and all docstrings are +removed. We recommend that default values and docstrings remain within +the type stub file so language servers can display this information to +developers. + +Library Interface +================= + +`PEP 561 <https://www.python.org/dev/peps/pep-0561/>`__ indicates that a +``py.typed`` marker file must be included in the package if the author +wishes to support type checking of their code. + +If a ``py.typed`` module is present, a type checker will treat all modules +within that package (i.e. all files that end in ``.py`` or ``.pyi``) as +importable unless the file name begins with an underscore. These modules +comprise the supported interface for the library. + +Each module exposes a set of symbols. Some of these symbols are +considered "private” — implementation details that are not part of the +library’s interface. Type checkers can use the following rules +to determine which symbols are visible outside of the package. + +- Symbols whose names begin with an underscore (but are not dunder + names) are considered private. +- Imported symbols are considered private by default. If they use the + ``import A as A`` (a redundant module alias), ``from X import A as A`` (a + redundant symbol alias), or ``from . import A`` forms, symbol ``A`` is + not private unless the name begins with an underscore. If a file + ``__init__.py`` uses form ``from .A import X``, symbol ``A`` is treated + likewise. If a wildcard import (of the form ``from X import *``) is + used, all symbols referenced by the wildcard are not private. +- A module can expose an ``__all__`` symbol at the module level that + provides a list of names that are considered part of the interface. + This overrides all other rules above, allowing imported symbols or + symbols whose names begin with an underscore to be included in the + interface. +- Local variables within a function (including nested functions) are + always considered private. + +The following idioms are supported for defining the values contained +within ``__all__``. These restrictions allow type checkers to statically +determine the value of ``__all__``. + +- ``__all__ = ('a', b')`` +- ``__all__ = ['a', b']`` +- ``__all__ += ['a', b']`` +- ``__all__ += submodule.__all__`` +- ``__all__.extend(['a', b'])`` +- ``__all__.extend(submodule.__all__)`` +- ``__all__.append('a')`` +- ``__all__.remove('a')`` + +Type Completeness +================= + +A “py.typed” library should aim to be type complete so that type +checking and inspection can work to their full extent. Here we say that a +library is “type complete” if all of the symbols +that comprise its interface have type annotations that refer to types +that are fully known. Private symbols are exempt. + +The following are best practice recommendations for how to define “type complete”: + +Classes: + +- All class variables, instance variables, and methods that are + “visible” (not overridden) are annotated and refer to known types +- If a class is a subclass of a generic class, type arguments are + provided for each generic type parameter, and these type arguments + are known types + +Functions and Methods: + +- All input parameters have type annotations that refer to known types +- The return parameter is annotated and refers to a known type +- The result of applying one or more decorators results in a known type + +Type Aliases: + +- All of the types referenced by the type alias are known + +Variables: + +- All variables have type annotations that refer to known types + +Type annotations can be omitted in a few specific cases where the type +is obvious from the context: + +- Constants that are assigned simple literal values + (e.g. ``RED = '#F00'`` or ``MAX_TIMEOUT = 50`` or + ``room_temperature: Final = 20``). A constant is a symbol that is + assigned only once and is either annotated with ``Final`` or is named + in all-caps. A constant that is not assigned a simple literal value + requires explicit annotations, preferably with a ``Final`` annotation + (e.g. ``WOODWINDS: Final[List[str]] = ['Oboe', 'Bassoon']``). +- Enum values within an Enum class do not require annotations because + they take on the type of the Enum class. +- Type aliases do not require annotations. A type alias is a symbol + that is defined at a module level with a single assignment where the + assigned value is an instantiable type, as opposed to a class + instance + (e.g. ``Foo = Callable[[Literal["a", "b"]], Union[int, str]]`` or + ``Bar = Optional[MyGenericClass[int]]``). +- The “self” parameter in an instance method and the “cls” parameter in + a class method do not require an explicit annotation. +- The return type for an ``__init__`` method does not need to be + specified, since it is always ``None``. +- The following module-level symbols do not require type annotations: + ``__all__``,\ ``__author__``, ``__copyright__``, ``__email__``, + ``__license__``, ``__title__``, ``__uri__``, ``__version__``. +- The following class-level symbols do not require type annotations: + ``__class__``, ``__dict__``, ``__doc__``, ``__module__``, + ``__slots__``. + +Examples of known and unknown types +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: python + + + # Variable with unknown type + a = [3, 4, 5] + + # Variable with known type + a: List[int] = [3, 4, 5] + + # Type alias with partially unknown type (because type + # arguments are missing for list and dict) + DictOrList = Union[list, dict] + + # Type alias with known type + DictOrList = Union[List[Any], Dict[str, Any]] + + # Generic type alias with known type + _T = TypeVar("_T") + DictOrList = Union[List[_T], Dict[str, _T]] + + # Function with known type + def func(a: Optional[int], b: Dict[str, float] = {}) -> None: + pass + + # Function with partially unknown type (because type annotations + # are missing for input parameters and return type) + def func(a, b): + pass + + # Function with partially unknown type (because of missing + # type args on Dict) + def func(a: int, b: Dict) -> None: + pass + + # Function with partially unknown type (because return type + # annotation is missing) + def func(a: int, b: Dict[str, float]): + pass + + # Decorator with partially unknown type (because type annotations + # are missing for input parameters and return type) + def my_decorator(func): + return func + + # Function with partially unknown type (because type is obscured + # by untyped decorator) + @my_decorator + def func(a: int) -> str: + pass + + + # Class with known type + class MyClass: + height: float = 2.0 + + def __init__(self, name: str, age: int): + self.age: int = age + + @property + def name(self) -> str: + ... + + # Class with partially unknown type + class MyClass: + # Missing type annotation for class variable + height = 2.0 + + # Missing input parameter annotations + def __init__(self, name, age): + # Missing type annotation for instance variable + self.age = age + + # Missing return type annotation + @property + def name(self): + ... + + # Class with partially unknown type + class BaseClass: + # Missing type annotation + height = 2.0 + + # Missing type annotation + def get_stuff(self): + ... + + # Class with known type (because it overrides all symbols + # exposed by BaseClass that have incomplete types) + class DerivedClass(BaseClass): + height: float + + def get_stuff(self) -> str: + ... + + # Class with partially unknown type because base class + # (dict) is generic, and type arguments are not specified. + class DictSubclass(dict): + pass + +Best Practices for Inlined Types +================================ + +Wide vs. Narrow Types +~~~~~~~~~~~~~~~~~~~~~ + +In type theory, when comparing two types that are related to each other, +the “wider” type is the one that is more general, and the “narrower” +type is more specific. For example, ``Sequence[str]`` is a wider type +than ``List[str]`` because all ``List`` objects are also ``Sequence`` +objects, but the converse is not true. A subclass is narrower than a +class it derives from. A union of types is wider than the individual +types that comprise the union. + +In general, a function input parameter should be annotated with the +widest possible type supported by the implementation. For example, if +the implementation requires the caller to provide an iterable collection +of strings, the parameter should be annotated as ``Iterable[str]``, not +as ``List[str]``. The latter type is narrower than necessary, so if a +user attempts to pass a tuple of strings (which is supported by the +implementation), a type checker will complain about a type +incompatibility. + +As a specific application of the “use the widest type possible” rule, +libraries should generally use immutable forms of container types +instead of mutable forms (unless the function needs to modify the +container). Use ``Sequence`` rather than ``List``, ``Mapping`` rather +than ``Dict``, etc. Immutable containers allow for more flexibility +because their type parameters are covariant rather than invariant. A +parameter that is typed as ``Sequence[Union[str, int]]`` can accept a +``List[int]``, ``Sequence[str]``, and a ``Sequence[int]``. But a +parameter typed as ``List[Union[str, int]]`` is much more restrictive +and accepts only a ``List[Union[str, int]]``. + +Overloads +~~~~~~~~~ + +If a function or method can return multiple different types and those +types can be determined based on the presence or types of certain +parameters, use the ``@overload`` mechanism defined in `PEP +484 <https://www.python.org/dev/peps/pep-0484/#id45>`__. When overloads +are used within a “.py” file, they must appear prior to the function +implementation, which should not have an ``@overload`` decorator. + +Keyword-only Parameters +~~~~~~~~~~~~~~~~~~~~~~~ + +If a function or method is intended to take parameters that are +specified only by name, use the keyword-only separator (``*``). + +.. code:: python + + def create_user(age: int, *, dob: Optional[date] = None): + ... + +Annotating Decorators +~~~~~~~~~~~~~~~~~~~~~ + +Decorators modify the behavior of a class or a function. Providing +annotations for decorators is straightforward if the decorator retains +the original signature of the decorated function. + +.. code:: python + + _F = TypeVar("_F", bound=Callable[..., Any]) + + def simple_decorator(_func: _F) -> _F: + """ + Simple decorators are invoked without parentheses like this: + @simple_decorator + def my_function(): ... + """ + ... + + def complex_decorator(*, mode: str) -> Callable[[_F], _F]: + """ + Complex decorators are invoked with arguments like this: + @complex_decorator(mode="easy") + def my_function(): ... + """ + ... + +Decorators that mutate the signature of the decorated function present +challenges for type annotations. The ``ParamSpec`` and ``Concatenate`` +mechanisms described in `PEP +612 <https://www.python.org/dev/peps/pep-0612/>`__ provide some help +here, but these are available only in Python 3.10 and newer. More +complex signature mutations may require type annotations that erase the +original signature, thus blinding type checkers and other tools that +provide signature assistance. As such, library authors are discouraged +from creating decorators that mutate function signatures in this manner. + +Generic Classes and Functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Classes and functions that can operate in a generic manner on various +types should declare themselves as generic using the mechanisms +described in `PEP 484 <https://www.python.org/dev/peps/pep-0484/>`__. +This includes the use of ``TypeVar`` symbols. Typically, a ``TypeVar`` +should be private to the file that declares it, and should therefore +begin with an underscore. + +Type Aliases +~~~~~~~~~~~~ + +Type aliases are symbols that refer to other types. Generic type aliases +(those that refer to unspecialized generic classes) are supported by +most type checkers. + +`PEP 613 <https://www.python.org/dev/peps/pep-0613/>`__ provides a way +to explicitly designate a symbol as a type alias using the new TypeAlias +annotation. + +.. code:: python + + # Simple type alias + FamilyPet = Union[Cat, Dog, GoldFish] + + # Generic type alias + ListOrTuple = Union[List[_T], Tuple[_T, ...]] + + # Recursive type alias + TreeNode = Union[LeafNode, List["TreeNode"]] + + # Explicit type alias using PEP 613 syntax + StrOrInt: TypeAlias = Union[str, int] + +Abstract Classes and Methods +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Classes that must be subclassed should derive from ``ABC``, and methods +or properties that must be overridden should be decorated with the +``@abstractmethod`` decorator. This allows type checkers to validate +that the required methods have been overridden and provide developers +with useful error messages when they are not. It is customary to +implement an abstract method by raising a ``NotImplementedError`` +exception. + +.. code:: python + + from abc import ABC, abstractmethod + + class Hashable(ABC): + @property + @abstractmethod + def hash_value(self) -> int: + """Subclasses must override""" + raise NotImplementedError() + + @abstractmethod + def print(self) -> str: + """Subclasses must override""" + raise NotImplementedError() + +Final Classes and Methods +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Classes that are not intended to be subclassed should be decorated as +``@final`` as described in `PEP +591 <https://www.python.org/dev/peps/pep-0591/>`__. The same decorator +can also be used to specify methods that cannot be overridden by +subclasses. + +Literals +~~~~~~~~ + +Type annotations should make use of the Literal type where appropriate, +as described in `PEP 586 <https://www.python.org/dev/peps/pep-0586/>`__. +Literals allow for more type specificity than their non-literal +counterparts. + +Constants +~~~~~~~~~ + +Constant values (those that are read-only) can be specified using the +Final annotation as described in `PEP +591 <https://www.python.org/dev/peps/pep-0591/>`__. + +Type checkers will also typically treat variables that are named using +all upper-case characters as constants. + +In both cases, it is OK to omit the declared type of a constant if it is +assigned a literal str, int, float, bool or None value. In such cases, +the type inference rules are clear and unambiguous, and adding a literal +type annotation would be redundant. + +.. code:: python + + # All-caps constant with inferred type + COLOR_FORMAT_RGB = "rgb" + + # All-caps constant with explicit type + COLOR_FORMAT_RGB: Literal["rgb"] = "rgb" + LATEST_VERSION: Tuple[int, int] = (4, 5) + + # Final variable with inferred type + ColorFormatRgb: Final = "rgb" + + # Final variable with explicit type + ColorFormatRgb: Final[Literal["rgb"]] = "rgb" + LATEST_VERSION: Final[Tuple[int, int]] = (4, 5) + +Typed Dictionaries, Data Classes, and Named Tuples +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If your library runs only on newer versions of Python, you are +encouraged to use some of the new type-friendly classes. + +NamedTuple (described in `PEP +484 <https://www.python.org/dev/peps/pep-0484/>`__) is preferred over +namedtuple. + +Data classes (described in `PEP +557 <https://www.python.org/dev/peps/pep-0557/>`__) is preferred over +untyped dictionaries. + +TypedDict (described in `PEP +589 <https://www.python.org/dev/peps/pep-0589/>`__) is preferred over +untyped dictionaries. + +Compatibility with Older Python Versions +======================================== + +Each new version of Python from 3.5 onward has introduced new typing +constructs. This presents a challenge for library authors who want to +maintain runtime compatibility with older versions of Python. This +section documents several techniques that can be used to add types while +maintaining backward compatibility. + +Quoted Annotations +~~~~~~~~~~~~~~~~~~ + +Type annotations for variables, parameters, and return types can be +placed in quotes. The Python interpreter will then ignore them, whereas +a type checker will interpret them as type annotations. + +.. code:: python + + # Older versions of Python do not support subscripting + # for the OrderedDict type, so the annotation must be + # enclosed in quotes. + def get_config(self) -> "OrderedDict[str, str]": + return self._config + +Type Comment Annotations +~~~~~~~~~~~~~~~~~~~~~~~~ + +Python 3.0 introduced syntax for parameter and return type annotations, +as specified in `PEP 484 <https://www.python.org/dev/peps/pep-0484/>`__. +Python 3.6 introduced support for variable type annotations, as +specified in `PEP 526 <https://www.python.org/dev/peps/pep-0526/>`__. + +If you need to support older versions of Python, type annotations can +still be provided as “type comments”. These comments take the form # +type: . + +.. code:: python + + class Foo: + # Variable type comments go at the end of the line + # where the variable is assigned. + timeout = None # type: Optional[int] + + # Function type comments can be specified on the + # line after the function signature. + def send_message(self, name, length): + # type: (str, int) -> None + ... + + # Function type comments can also specify the type + # of each parameter on its own line. + def receive_message( + self, + name, # type: str + length # type: int + ): + # type: () -> Message + ... + +typing_extensions +~~~~~~~~~~~~~~~~~ + +New type features that require runtime support are typically included in +the stdlib ``typing`` module. Where possible, these new features are +back-ported to a runtime library called ``typing_extensions`` that works +with older Python runtimes. + +TYPE_CHECKING +~~~~~~~~~~~~~ + +The ``typing`` module exposes a variable called ``TYPE_CHECKING`` which +has a value of False within the Python runtime but a value of True when +the type checker is performing its analysis. This allows type checking +statements to be conditionalized. + +Care should be taken when using ``TYPE_CHECKING`` because behavioral +changes between type checking and runtime could mask problems that the +type checker would otherwise catch. + +Non-Standard Type Behaviors +=========================== + +Type annotations provide a way to annotate typical type behaviors, but +some classes implement specialized, non-standard behaviors that cannot +be described using standard type annotations. For now, such types need +to be annotated as Any, which is unfortunate because the benefits of +static typing are lost. + +Docstrings +========== + +Docstrings should be provided for all classes, functions, and methods in +the interface. They should be formatted according to `PEP +257 <https://www.python.org/dev/peps/pep-0257/>`__. + +There is currently no single agreed-upon standard for function and +method docstrings, but several common variants have emerged. We +recommend using one of these variants. |