diff options
author | Greg Roodt <groodt@gmail.com> | 2022-09-25 06:04:15 +1000 |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-09-25 06:04:15 +1000 |
commit | f0efec5cf8c0ae16483ee677a09ec70737a01bf5 (patch) | |
tree | 79e958ef1e97b1941260b84283db615678a153dc /README.md | |
parent | a364fcac97b7ccadb5c5a703ec8ce2f2ecef0d97 (diff) | |
download | bazelbuild-rules_python-f0efec5cf8c0ae16483ee677a09ec70737a01bf5.tar.gz |
Standardise on pip_parse (#807)upstream/0.13.0
Diffstat (limited to 'README.md')
-rw-r--r-- | README.md | 81 |
1 files changed, 28 insertions, 53 deletions
@@ -7,8 +7,7 @@ This repository is the home of the core Python rules -- `py_library`, `py_binary`, `py_test`, and related symbols that provide the basis for Python -support in Bazel. It also contains packaging rules for integrating with PyPI -(`pip`). Documentation lives in the +support in Bazel. It also contains package installation rules for integrating with PyPI and other package indices. Documentation lives in the [`docs/`](https://github.com/bazelbuild/rules_python/tree/main/docs) directory and in the [Bazel Build Encyclopedia](https://docs.bazel.build/versions/master/be/python.html). @@ -24,7 +23,7 @@ Once they are fully migrated to rules_python, they may evolve at a different rate, but this repository will still follow [semantic versioning](https://semver.org). -The packaging rules (`pip_install`, etc.) are less stable. We may make breaking +The package installation rules (`pip_install`, `pip_parse` etc.) are less stable. We may make breaking changes as they evolve. This repository is maintained by the Bazel community. Neither Google, nor the @@ -101,14 +100,14 @@ py_binary( ) ``` -## Using the packaging rules +## Using the package installation rules Usage of the packaging rules involves two main steps. -1. [Installing `pip` dependencies](#installing-pip-dependencies) -2. [Consuming `pip` dependencies](#consuming-pip-dependencies) +1. [Installing third_party packages](#installing-third_party-packages) +2. [Using third_party packages as dependencies](#using-third_party-packages-as-dependencies) -The packaging rules create two kinds of repositories: A central external repo that holds +The package installation rules create two kinds of repositories: A central external repo that holds downloaded wheel files, and individual external repos for each wheel's extracted contents. Users only need to interact with the central external repo; the wheel repos are essentially an implementation detail. The central external repo provides a @@ -116,56 +115,13 @@ are essentially an implementation detail. The central external repo provides a `BUILD` files that translates a pip package name into the label of a `py_library` target in the appropriate wheel repo. -### Installing `pip` dependencies +### Installing third_party packages To add pip dependencies to your `WORKSPACE`, load the `pip_install` function, and call it to create the central external repo and individual wheel external repos. ```python -load("@rules_python//python:pip.bzl", "pip_install") - -# Create a central external repo, @my_deps, that contains Bazel targets for all the -# third-party packages specified in the requirements.txt file. -pip_install( - name = "my_deps", - requirements = "//path/to:requirements.txt", -) -``` - -Note that since `pip_install` is a repository rule and therefore executes pip at WORKSPACE-evaluation time, Bazel has no -information about the Python toolchain and cannot enforce that the interpreter -used to invoke pip matches the interpreter used to run `py_binary` targets. By -default, `pip_install` uses the system command `"python3"`. This can be overridden by passing the -`python_interpreter` attribute or `python_interpreter_target` attribute to `pip_install`. - -You can have multiple `pip_install`s in the same workspace. This will create multiple external repos that have no relation to -one another, and may result in downloading the same wheels multiple times. - -As with any repository rule, if you would like to ensure that `pip_install` is -re-executed in order to pick up a non-hermetic change to your environment (e.g., -updating your system `python` interpreter), you can force it to re-execute by running -`bazel sync --only [pip_install name]`. - -### Fetch `pip` dependencies lazily - -One pain point with `pip_install` is the need to download all dependencies resolved by -your requirements.txt before the bazel analysis phase can start. For large python monorepos -this can take a long time, especially on slow connections. - -`pip_parse` provides a solution to this problem. If you can provide a lock -file of all your python dependencies `pip_parse` will translate each requirement into its own external repository. -Bazel will only fetch/build wheels for the requirements in the subgraph of your build target. - -There are API differences between `pip_parse` and `pip_install`: -1. `pip_parse` requires a fully resolved lock file of your python dependencies. You can generate this by using the `compile_pip_requirements` rule, - running `pip-compile` directly, or using virtualenv and `pip freeze`. `pip_parse` uses a label argument called `requirements_lock` instead of - `requirements` to make this distinction clear. -2. `pip_parse` translates your requirements into a starlark macro called `install_deps`. You must call this macro in your WORKSPACE to - declare your dependencies. - - -```python load("@rules_python//python:pip.bzl", "pip_parse") # Create a central repo that knows about the dependencies needed from @@ -174,14 +130,33 @@ pip_parse( name = "my_deps", requirements_lock = "//path/to:requirements_lock.txt", ) - # Load the starlark macro which will define your dependencies. load("@my_deps//:requirements.bzl", "install_deps") # Call it to define repos for your requirements. install_deps() ``` -### Consuming `pip` dependencies +Note that since `pip_parse` is a repository rule and therefore executes pip at WORKSPACE-evaluation time, Bazel has no +information about the Python toolchain and cannot enforce that the interpreter +used to invoke pip matches the interpreter used to run `py_binary` targets. By +default, `pip_parse` uses the system command `"python3"`. This can be overridden by passing the +`python_interpreter` attribute or `python_interpreter_target` attribute to `pip_parse`. + +You can have multiple `pip_parse`s in the same workspace. This will create multiple external repos that have no relation to +one another, and may result in downloading the same wheels multiple times. + +As with any repository rule, if you would like to ensure that `pip_parse` is +re-executed in order to pick up a non-hermetic change to your environment (e.g., +updating your system `python` interpreter), you can force it to re-execute by running +`bazel sync --only [pip_parse name]`. + +Note: The `pip_install` rule is deprecated. `pip_parse` offers identical functionality and both `pip_install` +and `pip_parse` now have the same implementation. The name `pip_install` may be removed in a future version of the rules. +The maintainers have taken all reasonable efforts to faciliate a smooth transition, but some users of `pip_install` will +need to replace their existing `requirements.txt` with a fully resolved set of dependencies using a tool such as +`pip-tools` or the `compile_pip_requirements` repository rule. + +### Using third_party packages as dependencies Each extracted wheel repo contains a `py_library` target representing the wheel's contents. There are two ways to access this library. The |