diff options
author | Greg Roodt <groodt@gmail.com> | 2022-09-25 06:04:15 +1000 |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-09-25 06:04:15 +1000 |
commit | f0efec5cf8c0ae16483ee677a09ec70737a01bf5 (patch) | |
tree | 79e958ef1e97b1941260b84283db615678a153dc | |
parent | a364fcac97b7ccadb5c5a703ec8ce2f2ecef0d97 (diff) | |
download | bazelbuild-rules_python-upstream/0.13.0.tar.gz |
Standardise on pip_parse (#807)upstream/0.13.0
23 files changed, 139 insertions, 455 deletions
@@ -7,8 +7,7 @@ This repository is the home of the core Python rules -- `py_library`, `py_binary`, `py_test`, and related symbols that provide the basis for Python -support in Bazel. It also contains packaging rules for integrating with PyPI -(`pip`). Documentation lives in the +support in Bazel. It also contains package installation rules for integrating with PyPI and other package indices. Documentation lives in the [`docs/`](https://github.com/bazelbuild/rules_python/tree/main/docs) directory and in the [Bazel Build Encyclopedia](https://docs.bazel.build/versions/master/be/python.html). @@ -24,7 +23,7 @@ Once they are fully migrated to rules_python, they may evolve at a different rate, but this repository will still follow [semantic versioning](https://semver.org). -The packaging rules (`pip_install`, etc.) are less stable. We may make breaking +The package installation rules (`pip_install`, `pip_parse` etc.) are less stable. We may make breaking changes as they evolve. This repository is maintained by the Bazel community. Neither Google, nor the @@ -101,14 +100,14 @@ py_binary( ) ``` -## Using the packaging rules +## Using the package installation rules Usage of the packaging rules involves two main steps. -1. [Installing `pip` dependencies](#installing-pip-dependencies) -2. [Consuming `pip` dependencies](#consuming-pip-dependencies) +1. [Installing third_party packages](#installing-third_party-packages) +2. [Using third_party packages as dependencies](#using-third_party-packages-as-dependencies) -The packaging rules create two kinds of repositories: A central external repo that holds +The package installation rules create two kinds of repositories: A central external repo that holds downloaded wheel files, and individual external repos for each wheel's extracted contents. Users only need to interact with the central external repo; the wheel repos are essentially an implementation detail. The central external repo provides a @@ -116,56 +115,13 @@ are essentially an implementation detail. The central external repo provides a `BUILD` files that translates a pip package name into the label of a `py_library` target in the appropriate wheel repo. -### Installing `pip` dependencies +### Installing third_party packages To add pip dependencies to your `WORKSPACE`, load the `pip_install` function, and call it to create the central external repo and individual wheel external repos. ```python -load("@rules_python//python:pip.bzl", "pip_install") - -# Create a central external repo, @my_deps, that contains Bazel targets for all the -# third-party packages specified in the requirements.txt file. -pip_install( - name = "my_deps", - requirements = "//path/to:requirements.txt", -) -``` - -Note that since `pip_install` is a repository rule and therefore executes pip at WORKSPACE-evaluation time, Bazel has no -information about the Python toolchain and cannot enforce that the interpreter -used to invoke pip matches the interpreter used to run `py_binary` targets. By -default, `pip_install` uses the system command `"python3"`. This can be overridden by passing the -`python_interpreter` attribute or `python_interpreter_target` attribute to `pip_install`. - -You can have multiple `pip_install`s in the same workspace. This will create multiple external repos that have no relation to -one another, and may result in downloading the same wheels multiple times. - -As with any repository rule, if you would like to ensure that `pip_install` is -re-executed in order to pick up a non-hermetic change to your environment (e.g., -updating your system `python` interpreter), you can force it to re-execute by running -`bazel sync --only [pip_install name]`. - -### Fetch `pip` dependencies lazily - -One pain point with `pip_install` is the need to download all dependencies resolved by -your requirements.txt before the bazel analysis phase can start. For large python monorepos -this can take a long time, especially on slow connections. - -`pip_parse` provides a solution to this problem. If you can provide a lock -file of all your python dependencies `pip_parse` will translate each requirement into its own external repository. -Bazel will only fetch/build wheels for the requirements in the subgraph of your build target. - -There are API differences between `pip_parse` and `pip_install`: -1. `pip_parse` requires a fully resolved lock file of your python dependencies. You can generate this by using the `compile_pip_requirements` rule, - running `pip-compile` directly, or using virtualenv and `pip freeze`. `pip_parse` uses a label argument called `requirements_lock` instead of - `requirements` to make this distinction clear. -2. `pip_parse` translates your requirements into a starlark macro called `install_deps`. You must call this macro in your WORKSPACE to - declare your dependencies. - - -```python load("@rules_python//python:pip.bzl", "pip_parse") # Create a central repo that knows about the dependencies needed from @@ -174,14 +130,33 @@ pip_parse( name = "my_deps", requirements_lock = "//path/to:requirements_lock.txt", ) - # Load the starlark macro which will define your dependencies. load("@my_deps//:requirements.bzl", "install_deps") # Call it to define repos for your requirements. install_deps() ``` -### Consuming `pip` dependencies +Note that since `pip_parse` is a repository rule and therefore executes pip at WORKSPACE-evaluation time, Bazel has no +information about the Python toolchain and cannot enforce that the interpreter +used to invoke pip matches the interpreter used to run `py_binary` targets. By +default, `pip_parse` uses the system command `"python3"`. This can be overridden by passing the +`python_interpreter` attribute or `python_interpreter_target` attribute to `pip_parse`. + +You can have multiple `pip_parse`s in the same workspace. This will create multiple external repos that have no relation to +one another, and may result in downloading the same wheels multiple times. + +As with any repository rule, if you would like to ensure that `pip_parse` is +re-executed in order to pick up a non-hermetic change to your environment (e.g., +updating your system `python` interpreter), you can force it to re-execute by running +`bazel sync --only [pip_parse name]`. + +Note: The `pip_install` rule is deprecated. `pip_parse` offers identical functionality and both `pip_install` +and `pip_parse` now have the same implementation. The name `pip_install` may be removed in a future version of the rules. +The maintainers have taken all reasonable efforts to faciliate a smooth transition, but some users of `pip_install` will +need to replace their existing `requirements.txt` with a fully resolved set of dependencies using a tool such as +`pip-tools` or the `compile_pip_requirements` repository rule. + +### Using third_party packages as dependencies Each extracted wheel repo contains a `py_library` target representing the wheel's contents. There are two ways to access this library. The diff --git a/docs/pip.md b/docs/pip.md index 4853e52..f6d8430 100644 --- a/docs/pip.md +++ b/docs/pip.md @@ -73,69 +73,19 @@ Annotations to apply to the BUILD file content from package generated from a `pi pip_install(<a href="#pip_install-requirements">requirements</a>, <a href="#pip_install-name">name</a>, <a href="#pip_install-kwargs">kwargs</a>) </pre> -Accepts a `requirements.txt` file and installs the dependencies listed within. - -Those dependencies become available in a generated `requirements.bzl` file. - -This macro wraps the [`pip_repository`](./pip_repository.md) rule that invokes `pip`. -In your WORKSPACE file: +Accepts a locked/compiled requirements file and installs the dependencies listed within. ```python +load("@rules_python//python:pip.bzl", "pip_install") + pip_install( + name = "pip_deps", requirements = ":requirements.txt", ) -``` - -You can then reference installed dependencies from a `BUILD` file with: -```python -load("@pip//:requirements.bzl", "requirement") -py_library( - name = "bar", - ... - deps = [ - "//my/other:dep", - requirement("requests"), - requirement("numpy"), - ], -) -``` - -> Note that this convenience comes with a cost. -> Analysis of any BUILD file which loads the requirements helper in this way will -> cause an eager-fetch of all the pip dependencies, -> even if no python targets are requested to be built. -> In a multi-language repo, this may cause developers to fetch dependencies they don't need, -> so consider using the long form for dependencies if this happens. - -In addition to the `requirement` macro, which is used to access the `py_library` -target generated from a package's wheel, the generated `requirements.bzl` file contains -functionality for exposing [entry points][whl_ep] as `py_binary` targets. - -[whl_ep]: https://packaging.python.org/specifications/entry-points/ - -```python -load("@pip_deps//:requirements.bzl", "entry_point") - -alias( - name = "pip-compile", - actual = entry_point( - pkg = "pip-tools", - script = "pip-compile", - ), -) -``` - -Note that for packages whose name and script are the same, only the name of the package -is needed when calling the `entry_point` macro. - -```python -load("@pip_deps//:requirements.bzl", "entry_point") +load("@pip_deps//:requirements.bzl", "install_deps") -alias( - name = "flake8", - actual = entry_point("flake8"), -) +install_deps() ``` @@ -154,7 +104,7 @@ alias( ## pip_parse <pre> -pip_parse(<a href="#pip_parse-requirements_lock">requirements_lock</a>, <a href="#pip_parse-name">name</a>, <a href="#pip_parse-kwargs">kwargs</a>) +pip_parse(<a href="#pip_parse-requirements">requirements</a>, <a href="#pip_parse-requirements_lock">requirements_lock</a>, <a href="#pip_parse-name">name</a>, <a href="#pip_parse-kwargs">kwargs</a>) </pre> Accepts a locked/compiled requirements file and installs the dependencies listed within. @@ -247,7 +197,8 @@ See the example in rules_python/examples/pip_parse_vendored. | Name | Description | Default Value | | :-------------: | :-------------: | :-------------: | -| requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. Note that if your lockfile is platform-dependent, you can use the <code>requirements_[platform]</code> attributes. | none | +| requirements | Deprecated. See requirements_lock. | <code>None</code> | +| requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. Note that if your lockfile is platform-dependent, you can use the <code>requirements_[platform]</code> attributes. | <code>None</code> | | name | The name of the generated repository. The generated repositories containing each requirement will be of the form <name>_<requirement-name>. | <code>"pip_parsed_deps"</code> | | kwargs | Additional arguments to the [<code>pip_repository</code>](./pip_repository.md) repository rule. | none | diff --git a/docs/pip_repository.md b/docs/pip_repository.md index c66d8bf..875ea11 100644 --- a/docs/pip_repository.md +++ b/docs/pip_repository.md @@ -6,9 +6,9 @@ <pre> pip_repository(<a href="#pip_repository-name">name</a>, <a href="#pip_repository-annotations">annotations</a>, <a href="#pip_repository-download_only">download_only</a>, <a href="#pip_repository-enable_implicit_namespace_pkgs">enable_implicit_namespace_pkgs</a>, <a href="#pip_repository-environment">environment</a>, - <a href="#pip_repository-extra_pip_args">extra_pip_args</a>, <a href="#pip_repository-incremental">incremental</a>, <a href="#pip_repository-isolated">isolated</a>, <a href="#pip_repository-pip_data_exclude">pip_data_exclude</a>, <a href="#pip_repository-python_interpreter">python_interpreter</a>, - <a href="#pip_repository-python_interpreter_target">python_interpreter_target</a>, <a href="#pip_repository-quiet">quiet</a>, <a href="#pip_repository-repo_prefix">repo_prefix</a>, <a href="#pip_repository-requirements">requirements</a>, <a href="#pip_repository-requirements_darwin">requirements_darwin</a>, - <a href="#pip_repository-requirements_linux">requirements_linux</a>, <a href="#pip_repository-requirements_lock">requirements_lock</a>, <a href="#pip_repository-requirements_windows">requirements_windows</a>, <a href="#pip_repository-timeout">timeout</a>) + <a href="#pip_repository-extra_pip_args">extra_pip_args</a>, <a href="#pip_repository-isolated">isolated</a>, <a href="#pip_repository-pip_data_exclude">pip_data_exclude</a>, <a href="#pip_repository-python_interpreter">python_interpreter</a>, + <a href="#pip_repository-python_interpreter_target">python_interpreter_target</a>, <a href="#pip_repository-quiet">quiet</a>, <a href="#pip_repository-repo_prefix">repo_prefix</a>, <a href="#pip_repository-requirements_darwin">requirements_darwin</a>, <a href="#pip_repository-requirements_linux">requirements_linux</a>, + <a href="#pip_repository-requirements_lock">requirements_lock</a>, <a href="#pip_repository-requirements_windows">requirements_windows</a>, <a href="#pip_repository-timeout">timeout</a>) </pre> A rule for importing `requirements.txt` dependencies into Bazel. @@ -62,14 +62,12 @@ py_binary( | enable_implicit_namespace_pkgs | If true, disables conversion of native namespace packages into pkg-util style namespace packages. When set all py_binary and py_test targets must specify either <code>legacy_create_init=False</code> or the global Bazel option <code>--incompatible_default_to_explicit_init_py</code> to prevent <code>__init__.py</code> being automatically generated in every directory.<br><br>This option is required to support some packages which cannot handle the conversion to pkg-util style. | Boolean | optional | False | | environment | Environment variables to set in the pip subprocess. Can be used to set common variables such as <code>http_proxy</code>, <code>https_proxy</code> and <code>no_proxy</code> Note that pip is run with "--isolated" on the CLI so PIP_<VAR>_<NAME> style env vars are ignored, but env vars that control requests and urllib3 can be passed. | <a href="https://bazel.build/docs/skylark/lib/dict.html">Dictionary: String -> String</a> | optional | {} | | extra_pip_args | Extra arguments to pass on to pip. Must not contain spaces. | List of strings | optional | [] | -| incremental | Create the repository in incremental mode. | Boolean | optional | False | | isolated | Whether or not to pass the [--isolated](https://pip.pypa.io/en/stable/cli/pip/#cmdoption-isolated) flag to the underlying pip command. Alternatively, the <code>RULES_PYTHON_PIP_ISOLATED</code> enviornment varaible can be used to control this flag. | Boolean | optional | True | | pip_data_exclude | Additional data exclusion parameters to add to the pip packages BUILD file. | List of strings | optional | [] | | python_interpreter | The python interpreter to use. This can either be an absolute path or the name of a binary found on the host's <code>PATH</code> environment variable. If no value is set <code>python3</code> is defaulted for Unix systems and <code>python.exe</code> for Windows. | String | optional | "" | | python_interpreter_target | If you are using a custom python interpreter built by another repository rule, use this attribute to specify its BUILD target. This allows pip_repository to invoke pip using the same interpreter as your toolchain. If set, takes precedence over python_interpreter. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None | | quiet | If True, suppress printing stdout and stderr output to the terminal. | Boolean | optional | True | -| repo_prefix | Prefix for the generated packages. For non-incremental mode the packages will be of the form<br><br>@<name>//<prefix><sanitized-package-name>/...<br><br>For incremental mode the packages will be of the form<br><br>@<prefix><sanitized-package-name>//... | String | optional | "" | -| requirements | A 'requirements.txt' pip requirements file. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None | +| repo_prefix | Prefix for the generated packages will be of the form<br><br>@<prefix><sanitized-package-name>//... | String | optional | "" | | requirements_darwin | Override the requirements_lock attribute when the host platform is Mac OS | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None | | requirements_linux | Override the requirements_lock attribute when the host platform is Linux | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None | | requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None | @@ -108,7 +106,7 @@ Instantiated from pip_repository and inherits config options from there. | python_interpreter_target | If you are using a custom python interpreter built by another repository rule, use this attribute to specify its BUILD target. This allows pip_repository to invoke pip using the same interpreter as your toolchain. If set, takes precedence over python_interpreter. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None | | quiet | If True, suppress printing stdout and stderr output to the terminal. | Boolean | optional | True | | repo | Pointer to parent repo name. Used to make these rules rerun if the parent repo changes. | String | required | | -| repo_prefix | Prefix for the generated packages. For non-incremental mode the packages will be of the form<br><br>@<name>//<prefix><sanitized-package-name>/...<br><br>For incremental mode the packages will be of the form<br><br>@<prefix><sanitized-package-name>//... | String | optional | "" | +| repo_prefix | Prefix for the generated packages will be of the form<br><br>@<prefix><sanitized-package-name>//... | String | optional | "" | | requirement | Python requirement string describing the package to make available | String | required | | | timeout | Timeout (in seconds) on the rule's execution duration. | Integer | optional | 600 | diff --git a/examples/BUILD b/examples/BUILD index 41dd875..ee4d7e4 100644 --- a/examples/BUILD +++ b/examples/BUILD @@ -39,10 +39,5 @@ bazel_integration_test( ) bazel_integration_test( - name = "relative_requirements_example", - timeout = "long", -) - -bazel_integration_test( name = "bzlmod_example", ) diff --git a/examples/pip_install/BUILD b/examples/pip_install/BUILD index ad983b2..35f5a93 100644 --- a/examples/pip_install/BUILD +++ b/examples/pip_install/BUILD @@ -88,9 +88,9 @@ py_test( genquery( name = "yamllint_lib_by_version", expression = """ - attr("tags", "\\bpypi_version=1.26.3\\b", "@pip//pypi__yamllint") + attr("tags", "\\bpypi_version=1.26.3\\b", "@pip_yamllint//:pkg") intersect - attr("tags", "\\bpypi_name=yamllint\\b", "@pip//pypi__yamllint") + attr("tags", "\\bpypi_name=yamllint\\b", "@pip_yamllint//:pkg") """, scope = [requirement("yamllint")], ) @@ -99,7 +99,7 @@ write_file( name = "write_expected", out = "expected", content = [ - "@pip//pypi__yamllint:pypi__yamllint", + "@pip_yamllint//:pkg", "", ], ) diff --git a/examples/pip_install/WORKSPACE b/examples/pip_install/WORKSPACE index 0b33a2b..f63d928 100644 --- a/examples/pip_install/WORKSPACE +++ b/examples/pip_install/WORKSPACE @@ -57,6 +57,11 @@ pip_install( requirements = "//:requirements.txt", ) +load("@pip//:requirements.bzl", "install_deps") + +# Initialize repositories for all packages in requirements.txt. +install_deps() + # You could optionally use an in-build, compiled python interpreter as a toolchain, # and also use it to execute pip. # diff --git a/examples/pip_install/pip_install_test.py b/examples/pip_install/pip_install_test.py index 6092768..9fe51fa 100644 --- a/examples/pip_install/pip_install_test.py +++ b/examples/pip_install/pip_install_test.py @@ -37,11 +37,11 @@ class PipInstallTest(unittest.TestCase): self.assertListEqual( env.split(" "), [ - "external/pip/pypi__s3cmd/data/share/doc/packages/s3cmd/INSTALL.md", - "external/pip/pypi__s3cmd/data/share/doc/packages/s3cmd/LICENSE", - "external/pip/pypi__s3cmd/data/share/doc/packages/s3cmd/NEWS", - "external/pip/pypi__s3cmd/data/share/doc/packages/s3cmd/README.md", - "external/pip/pypi__s3cmd/data/share/man/man1/s3cmd.1", + "external/pip_s3cmd/data/share/doc/packages/s3cmd/INSTALL.md", + "external/pip_s3cmd/data/share/doc/packages/s3cmd/LICENSE", + "external/pip_s3cmd/data/share/doc/packages/s3cmd/NEWS", + "external/pip_s3cmd/data/share/doc/packages/s3cmd/README.md", + "external/pip_s3cmd/data/share/man/man1/s3cmd.1", ], ) @@ -51,13 +51,13 @@ class PipInstallTest(unittest.TestCase): self.assertListEqual( env.split(" "), [ - "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/DESCRIPTION.rst", - "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/INSTALLER", - "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/METADATA", - "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/RECORD", - "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/WHEEL", - "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/metadata.json", - "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/top_level.txt", + "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/DESCRIPTION.rst", + "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/INSTALLER", + "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/METADATA", + "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/RECORD", + "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/WHEEL", + "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/metadata.json", + "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/top_level.txt", ], ) diff --git a/examples/pip_repository_annotations/BUILD b/examples/pip_repository_annotations/BUILD index 8c69c40..4fd124e 100644 --- a/examples/pip_repository_annotations/BUILD +++ b/examples/pip_repository_annotations/BUILD @@ -27,7 +27,7 @@ py_test( py_test( name = "pip_install_annotations_test", srcs = ["pip_repository_annotations_test.py"], - env = {"WHEEL_PKG_DIR": "pip_installed/pypi__wheel"}, + env = {"WHEEL_PKG_DIR": "pip_installed_wheel"}, main = "pip_repository_annotations_test.py", deps = [ requirement("wheel"), diff --git a/examples/pip_repository_annotations/WORKSPACE b/examples/pip_repository_annotations/WORKSPACE index 8ee885d..aeea842 100644 --- a/examples/pip_repository_annotations/WORKSPACE +++ b/examples/pip_repository_annotations/WORKSPACE @@ -54,9 +54,9 @@ pip_parse( requirements_lock = "//:requirements.txt", ) -load("@pip_parsed//:requirements.bzl", "install_deps") +load("@pip_parsed//:requirements.bzl", install_pip_parse_deps = "install_deps") -install_deps() +install_pip_parse_deps() # For a more thorough example of `pip_install`. See `@rules_python//examples/pip_install` pip_install( @@ -65,3 +65,7 @@ pip_install( python_interpreter_target = interpreter, requirements = "//:requirements.txt", ) + +load("@pip_installed//:requirements.bzl", install_pip_install_deps = "install_deps") + +install_pip_install_deps() diff --git a/examples/relative_requirements/BUILD b/examples/relative_requirements/BUILD deleted file mode 100644 index d24ee5f..0000000 --- a/examples/relative_requirements/BUILD +++ /dev/null @@ -1,10 +0,0 @@ -load("@pip//:requirements.bzl", "requirement") -load("@rules_python//python:defs.bzl", "py_test") - -py_test( - name = "main", - srcs = ["main.py"], - deps = [ - requirement("relative_package_name"), - ], -) diff --git a/examples/relative_requirements/README.md b/examples/relative_requirements/README.md deleted file mode 100644 index 4b9258e..0000000 --- a/examples/relative_requirements/README.md +++ /dev/null @@ -1,4 +0,0 @@ -# relative_requirements example - -This example shows how to use pip to fetch relative dependencies from a requirements.txt file, -then use them in BUILD files as dependencies of Bazel targets. diff --git a/examples/relative_requirements/WORKSPACE b/examples/relative_requirements/WORKSPACE deleted file mode 100644 index 4ae91c3..0000000 --- a/examples/relative_requirements/WORKSPACE +++ /dev/null @@ -1,21 +0,0 @@ -workspace(name = "example_repo") - -local_repository( - name = "rules_python", - path = "../..", -) - -load("@rules_python//python:repositories.bzl", "python_register_toolchains") - -python_register_toolchains( - name = "python39", - python_version = "3.9", -) - -load("@python39//:defs.bzl", "interpreter") -load("@rules_python//python:pip.bzl", "pip_install") - -pip_install( - python_interpreter_target = interpreter, - requirements = "//:requirements.txt", -) diff --git a/examples/relative_requirements/main.py b/examples/relative_requirements/main.py deleted file mode 100644 index b8ac021..0000000 --- a/examples/relative_requirements/main.py +++ /dev/null @@ -1,5 +0,0 @@ -import relative_package_name - -if __name__ == "__main__": - # Run a function from the relative package - print(relative_package_name.test()) diff --git a/examples/relative_requirements/relative_package/relative_package_name/__init__.py b/examples/relative_requirements/relative_package/relative_package_name/__init__.py deleted file mode 100644 index c031192..0000000 --- a/examples/relative_requirements/relative_package/relative_package_name/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -def test(): - return True diff --git a/examples/relative_requirements/relative_package/setup.py b/examples/relative_requirements/relative_package/setup.py deleted file mode 100644 index 052b519..0000000 --- a/examples/relative_requirements/relative_package/setup.py +++ /dev/null @@ -1,7 +0,0 @@ -from setuptools import setup - -setup( - name="relative_package_name", - version="1.0.0", - packages=["relative_package_name"], -) diff --git a/examples/relative_requirements/requirements.txt b/examples/relative_requirements/requirements.txt deleted file mode 100644 index 9a81317..0000000 --- a/examples/relative_requirements/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -./relative_package diff --git a/python/pip.bzl b/python/pip.bzl index 954317f..dfafefe 100644 --- a/python/pip.bzl +++ b/python/pip.bzl @@ -21,69 +21,19 @@ compile_pip_requirements = _compile_pip_requirements package_annotation = _package_annotation def pip_install(requirements = None, name = "pip", **kwargs): - """Accepts a `requirements.txt` file and installs the dependencies listed within. - - Those dependencies become available in a generated `requirements.bzl` file. - - This macro wraps the [`pip_repository`](./pip_repository.md) rule that invokes `pip`. - In your WORKSPACE file: + """Accepts a locked/compiled requirements file and installs the dependencies listed within. ```python + load("@rules_python//python:pip.bzl", "pip_install") + pip_install( + name = "pip_deps", requirements = ":requirements.txt", ) - ``` - - You can then reference installed dependencies from a `BUILD` file with: - - ```python - load("@pip//:requirements.bzl", "requirement") - py_library( - name = "bar", - ... - deps = [ - "//my/other:dep", - requirement("requests"), - requirement("numpy"), - ], - ) - ``` - - > Note that this convenience comes with a cost. - > Analysis of any BUILD file which loads the requirements helper in this way will - > cause an eager-fetch of all the pip dependencies, - > even if no python targets are requested to be built. - > In a multi-language repo, this may cause developers to fetch dependencies they don't need, - > so consider using the long form for dependencies if this happens. - - In addition to the `requirement` macro, which is used to access the `py_library` - target generated from a package's wheel, the generated `requirements.bzl` file contains - functionality for exposing [entry points][whl_ep] as `py_binary` targets. - - [whl_ep]: https://packaging.python.org/specifications/entry-points/ - - ```python - load("@pip_deps//:requirements.bzl", "entry_point") - alias( - name = "pip-compile", - actual = entry_point( - pkg = "pip-tools", - script = "pip-compile", - ), - ) - ``` - - Note that for packages whose name and script are the same, only the name of the package - is needed when calling the `entry_point` macro. - - ```python - load("@pip_deps//:requirements.bzl", "entry_point") + load("@pip_deps//:requirements.bzl", "install_deps") - alias( - name = "flake8", - actual = entry_point("flake8"), - ) + install_deps() ``` Args: @@ -92,17 +42,11 @@ def pip_install(requirements = None, name = "pip", **kwargs): **kwargs (dict): Additional arguments to the [`pip_repository`](./pip_repository.md) repository rule. """ - # Just in case our dependencies weren't already fetched - pip_install_dependencies() - - pip_repository( - name = name, - requirements = requirements, - repo_prefix = "pypi__", - **kwargs - ) + # buildifier: disable=print + print("pip_install is deprecated. Please switch to pip_parse. pip_install will be removed in a future release.") + pip_parse(requirements = requirements, name = name, **kwargs) -def pip_parse(requirements_lock, name = "pip_parsed_deps", **kwargs): +def pip_parse(requirements = None, requirements_lock = None, name = "pip_parsed_deps", **kwargs): """Accepts a locked/compiled requirements file and installs the dependencies listed within. Those dependencies become available in a generated `requirements.bzl` file. @@ -195,6 +139,7 @@ def pip_parse(requirements_lock, name = "pip_parsed_deps", **kwargs): fetched/built only for the targets specified by 'build/run/test'. Note that if your lockfile is platform-dependent, you can use the `requirements_[platform]` attributes. + requirements (Label): Deprecated. See requirements_lock. name (str, optional): The name of the generated repository. The generated repositories containing each requirement will be of the form <name>_<requirement-name>. **kwargs (dict): Additional arguments to the [`pip_repository`](./pip_repository.md) repository rule. @@ -203,10 +148,14 @@ def pip_parse(requirements_lock, name = "pip_parsed_deps", **kwargs): # Just in case our dependencies weren't already fetched pip_install_dependencies() + # Temporary compatibility shim. + # pip_install was previously document to use requirements while pip_parse was using requirements_lock. + # We would prefer everyone move to using requirements_lock, but we maintain a temporary shim. + reqs_to_use = requirements_lock if requirements_lock else requirements + pip_repository( name = name, - requirements_lock = requirements_lock, + requirements_lock = reqs_to_use, repo_prefix = "{}_".format(name), - incremental = True, **kwargs ) diff --git a/python/pip_install/extract_wheels/BUILD b/python/pip_install/extract_wheels/BUILD index 158d34b..bc11885 100644 --- a/python/pip_install/extract_wheels/BUILD +++ b/python/pip_install/extract_wheels/BUILD @@ -9,7 +9,6 @@ py_library( "arguments.py", "bazel.py", "extract_single_wheel.py", - "extract_wheels.py", "namespace_pkgs.py", "parse_requirements_to_bzl.py", "requirements.py", @@ -22,14 +21,6 @@ py_library( ) py_binary( - name = "extract_wheels", - srcs = [ - "extract_wheels.py", - ], - deps = [":lib"], -) - -py_binary( name = "extract_single_wheel", srcs = [ "extract_single_wheel.py", diff --git a/python/pip_install/extract_wheels/extract_single_wheel.py b/python/pip_install/extract_wheels/extract_single_wheel.py index a7cc672..9c44eff 100644 --- a/python/pip_install/extract_wheels/extract_single_wheel.py +++ b/python/pip_install/extract_wheels/extract_single_wheel.py @@ -8,11 +8,32 @@ from tempfile import NamedTemporaryFile from python.pip_install.extract_wheels import arguments, bazel, requirements from python.pip_install.extract_wheels.annotation import annotation_from_str_path -from python.pip_install.extract_wheels.extract_wheels import ( - configure_reproducible_wheels, -) +def configure_reproducible_wheels() -> None: + """Modifies the environment to make wheel building reproducible. + Wheels created from sdists are not reproducible by default. We can however workaround this by + patching in some configuration with environment variables. + """ + + # wheel, by default, enables debug symbols in GCC. This incidentally captures the build path in the .so file + # We can override this behavior by disabling debug symbols entirely. + # https://github.com/pypa/pip/issues/6505 + if "CFLAGS" in os.environ: + os.environ["CFLAGS"] += " -g0" + else: + os.environ["CFLAGS"] = "-g0" + + # set SOURCE_DATE_EPOCH to 1980 so that we can use python wheels + # https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#python-setuppy-bdist_wheel-cannot-create-whl + if "SOURCE_DATE_EPOCH" not in os.environ: + os.environ["SOURCE_DATE_EPOCH"] = "315532800" + + # Python wheel metadata files can be unstable. + # See https://bitbucket.org/pypa/wheel/pull-requests/74/make-the-output-of-metadata-files/diff + if "PYTHONHASHSEED" not in os.environ: + os.environ["PYTHONHASHSEED"] = "0" + def main() -> None: parser = argparse.ArgumentParser( description="Build and/or fetch a single wheel based on the requirement passed in" diff --git a/python/pip_install/extract_wheels/extract_wheels.py b/python/pip_install/extract_wheels/extract_wheels.py deleted file mode 100644 index 2addaf8..0000000 --- a/python/pip_install/extract_wheels/extract_wheels.py +++ /dev/null @@ -1,132 +0,0 @@ -"""extract_wheels - -extract_wheels resolves and fetches artifacts transitively from the Python Package Index (PyPI) based on a -requirements.txt. It generates the required BUILD files to consume these packages as Python libraries. - -Under the hood, it depends on the `pip wheel` command to do resolution, download, and compilation into wheels. -""" -import argparse -import glob -import os -import pathlib -import subprocess -import sys - -from python.pip_install.extract_wheels import ( - annotation, - arguments, - bazel, - requirements, - wheel, -) - - -def configure_reproducible_wheels() -> None: - """Modifies the environment to make wheel building reproducible. - - Wheels created from sdists are not reproducible by default. We can however workaround this by - patching in some configuration with environment variables. - """ - - # wheel, by default, enables debug symbols in GCC. This incidentally captures the build path in the .so file - # We can override this behavior by disabling debug symbols entirely. - # https://github.com/pypa/pip/issues/6505 - if "CFLAGS" in os.environ: - os.environ["CFLAGS"] += " -g0" - else: - os.environ["CFLAGS"] = "-g0" - - # set SOURCE_DATE_EPOCH to 1980 so that we can use python wheels - # https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#python-setuppy-bdist_wheel-cannot-create-whl - if "SOURCE_DATE_EPOCH" not in os.environ: - os.environ["SOURCE_DATE_EPOCH"] = "315532800" - - # Python wheel metadata files can be unstable. - # See https://bitbucket.org/pypa/wheel/pull-requests/74/make-the-output-of-metadata-files/diff - if "PYTHONHASHSEED" not in os.environ: - os.environ["PYTHONHASHSEED"] = "0" - - -def main() -> None: - """Main program. - - Exits zero on successful program termination, non-zero otherwise. - """ - - configure_reproducible_wheels() - - parser = argparse.ArgumentParser( - description="Resolve and fetch artifacts transitively from PyPI" - ) - parser.add_argument( - "--requirements", - action="store", - required=True, - help="Path to requirements.txt from where to install dependencies", - ) - parser.add_argument( - "--annotations", - type=annotation.annotations_map_from_str_path, - help="A json encoded file containing annotations for rendered packages.", - ) - arguments.parse_common_args(parser) - args = parser.parse_args() - deserialized_args = dict(vars(args)) - arguments.deserialize_structured_args(deserialized_args) - - # Pip is run with the working directory changed to the folder containing the requirements.txt file, to allow for - # relative requirements to be correctly resolved. The --wheel-dir is therefore required to be repointed back to the - # current calling working directory (the repo root in .../external/name), where the wheel files should be written to - pip_args = ( - [sys.executable, "-m", "pip"] - + (["--isolated"] if args.isolated else []) - + ["download" if args.download_only else "wheel", "-r", args.requirements] - + ["--wheel-dir", os.getcwd()] - + deserialized_args["extra_pip_args"] - ) - - env = os.environ.copy() - env.update(deserialized_args["environment"]) - - # Assumes any errors are logged by pip so do nothing. This command will fail if pip fails - subprocess.run( - pip_args, - check=True, - env=env, - cwd=str(pathlib.Path(args.requirements).parent.resolve()), - ) - - extras = requirements.parse_extras(args.requirements) - - repo_label = "@%s" % args.repo - - # Locate all wheels - wheels = [whl for whl in glob.glob("*.whl")] - - # Collect all annotations - reqs = {whl: wheel.Wheel(whl).name for whl in wheels} - annotations = args.annotations.collect(reqs.values()) - - targets = [ - '"{}{}"'.format( - repo_label, - bazel.extract_wheel( - wheel_file=whl, - extras=extras, - pip_data_exclude=deserialized_args["pip_data_exclude"], - enable_implicit_namespace_pkgs=args.enable_implicit_namespace_pkgs, - repo_prefix=args.repo_prefix, - annotation=annotations.get(name), - ), - ) - for whl, name in reqs.items() - ] - - with open("requirements.bzl", "w") as requirement_file: - requirement_file.write( - bazel.generate_requirements_file_contents(repo_label, targets) - ) - - -if __name__ == "__main__": - main() diff --git a/python/pip_install/pip_repository.bzl b/python/pip_install/pip_repository.bzl index d729ae9..bc7da73 100644 --- a/python/pip_install/pip_repository.bzl +++ b/python/pip_install/pip_repository.bzl @@ -222,8 +222,7 @@ def _locked_requirements(rctx): requirements_txt = rctx.attr.requirements_windows if not requirements_txt: fail("""\ -Incremental mode requires a requirements_lock attribute be specified, -or a platform-specific lockfile using one of the requirements_* attributes. +A requirements_lock attribute must be specified, or a platform-specific lockfile using one of the requirements_* attributes. """) return requirements_txt @@ -235,40 +234,28 @@ def _pip_repository_impl(rctx): annotations_file = rctx.path("annotations.json") rctx.file(annotations_file, json.encode_indent(annotations, indent = " " * 4)) - if rctx.attr.incremental: - requirements_txt = _locked_requirements(rctx) - args = [ - python_interpreter, - "-m", - "python.pip_install.extract_wheels.parse_requirements_to_bzl", - "--requirements_lock", - rctx.path(requirements_txt), - "--requirements_lock_label", - str(requirements_txt), - # pass quiet and timeout args through to child repos. - "--quiet", - str(rctx.attr.quiet), - "--timeout", - str(rctx.attr.timeout), - "--annotations", - annotations_file, - ] + requirements_txt = _locked_requirements(rctx) + args = [ + python_interpreter, + "-m", + "python.pip_install.extract_wheels.parse_requirements_to_bzl", + "--requirements_lock", + rctx.path(requirements_txt), + "--requirements_lock_label", + str(requirements_txt), + # pass quiet and timeout args through to child repos. + "--quiet", + str(rctx.attr.quiet), + "--timeout", + str(rctx.attr.timeout), + "--annotations", + annotations_file, + ] - args += ["--python_interpreter", _get_python_interpreter_attr(rctx)] - if rctx.attr.python_interpreter_target: - args += ["--python_interpreter_target", str(rctx.attr.python_interpreter_target)] - progress_message = "Parsing requirements to starlark" - else: - args = [ - python_interpreter, - "-m", - "python.pip_install.extract_wheels.extract_wheels", - "--requirements", - rctx.path(rctx.attr.requirements), - "--annotations", - annotations_file, - ] - progress_message = "Extracting wheels" + args += ["--python_interpreter", _get_python_interpreter_attr(rctx)] + if rctx.attr.python_interpreter_target: + args += ["--python_interpreter_target", str(rctx.attr.python_interpreter_target)] + progress_message = "Parsing requirements to starlark" args += ["--repo", rctx.attr.name, "--repo-prefix", rctx.attr.repo_prefix] args = _parse_optional_attrs(rctx, args) @@ -361,12 +348,7 @@ python_interpreter. ), "repo_prefix": attr.string( doc = """ -Prefix for the generated packages. For non-incremental mode the -packages will be of the form - -@<name>//<prefix><sanitized-package-name>/... - -For incremental mode the packages will be of the form +Prefix for the generated packages will be of the form @<prefix><sanitized-package-name>//... """, @@ -387,14 +369,6 @@ pip_repository_attrs = { "annotations": attr.string_dict( doc = "Optional annotations to apply to packages", ), - "incremental": attr.bool( - default = False, - doc = "Create the repository in incremental mode.", - ), - "requirements": attr.label( - allow_single_file = True, - doc = "A 'requirements.txt' pip requirements file.", - ), "requirements_darwin": attr.label( allow_single_file = True, doc = "Override the requirements_lock attribute when the host platform is Mac OS", diff --git a/python/pip_install/private/srcs.bzl b/python/pip_install/private/srcs.bzl index bdd76b1..e42bb8e 100644 --- a/python/pip_install/private/srcs.bzl +++ b/python/pip_install/private/srcs.bzl @@ -12,7 +12,6 @@ PIP_INSTALL_PY_SRCS = [ "@rules_python//python/pip_install/extract_wheels:arguments.py", "@rules_python//python/pip_install/extract_wheels:bazel.py", "@rules_python//python/pip_install/extract_wheels:extract_single_wheel.py", - "@rules_python//python/pip_install/extract_wheels:extract_wheels.py", "@rules_python//python/pip_install/extract_wheels:namespace_pkgs.py", "@rules_python//python/pip_install/extract_wheels:parse_requirements_to_bzl.py", "@rules_python//python/pip_install/extract_wheels:requirements.py", diff --git a/tests/pip_repository_entry_points/WORKSPACE b/tests/pip_repository_entry_points/WORKSPACE index 07a5d3a..dd80db4 100644 --- a/tests/pip_repository_entry_points/WORKSPACE +++ b/tests/pip_repository_entry_points/WORKSPACE @@ -24,9 +24,9 @@ pip_parse( requirements_lock = "//:requirements.txt", ) -load("@pip_parsed//:requirements.bzl", "install_deps") +load("@pip_parsed//:requirements.bzl", install_pip_parse_deps = "install_deps") -install_deps() +install_pip_parse_deps() # For a more thorough example of `pip_install`. See `@rules_python//examples/pip_install` pip_install( @@ -34,3 +34,7 @@ pip_install( python_interpreter_target = interpreter, requirements = "//:requirements.txt", ) + +load("@pip_installed//:requirements.bzl", install_pip_install_deps = "install_deps") + +install_pip_install_deps() |