diff options
| author | mattmoor <mattmoor@google.com> | 2017-09-15 14:12:03 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2017-09-15 14:12:03 -0700 |
| commit | 7f4cc9244dac7637d514e4f86364507681dda37e (patch) | |
| tree | 60ba189ef3271ac9e79ab0b1f18963125b4510ab | |
| parent | 9160fc7367be8db56ae90c581bd0ceebda969e24 (diff) | |
| download | bazelbuild-rules_python-7f4cc9244dac7637d514e4f86364507681dda37e.tar.gz | |
PIP dependency support (#1)
* Check in an initial version of the pip rules.
This is broken off of my prototype repository, and there are a handful of TODOs left to resolve.
* Remove some vestigial convenience code.
Elaborate on why the TODO to rewrite {pip,whl}.sh as Python is harder than it looks.
* Incorporate Code Review Feedback
* First cut at skydoc-based documentation.
* Move whl.sh into a Python script that has unit testing.
* Migrate pip.sh into piptool.py
This migrates the logic from pip.sh into piptool.py, which should improve portability by removing the bash dependency.
This also has the beginnings of wrapping piptool as a closed redistributable that doesn't rely on a system-installed copy of PIP, but instead uses these rules to pull pip into a PAR bundle. Besides needing to work out the details of releasing and redistributing the PAR, we have two unresolved issues:
* When bundled as a PAR (vs. py_binary), piptool seems to pick up the system-installed version of pip.
* When bundled as a PAR, piptool sometimes sees cert issues resolving requirements (similar to what we see with httplib2).
* Address the cert issue in piptool as a PAR.
With this change I am able to build/test my PR on my macbook without pip installed. The only additional change I have locally is to switch from running piptool.py as a simple .py file to downloading/using a PAR built from this change.
I believe we still have the problem that the .par picks the host's version of pip instead of our embedded copy, but I haven't reverified that issue still exists (this just does nothing to address that issue).
* Fix assorted buildifier issues.
* Fix a typo in docs
* Incorporate @duggelz code review feedback.
* Incorporate review feedback
move python tools under a top-level rules_python package
simplify version_test
* Adopt a canonical naming format for imported whl_library rules.
`whl_library` rules generated by `pip_import` are now named as:
```
pypi__{distribution}_{version}
```
Substituting illegal characters (e.g. `-`, `.`) with underscores.
* Move the piptool dependency into a .PAR file.
Add a script for updating the tools and document this and the docs script.
* Fix buildifier issues.
37 files changed, 2118 insertions, 12 deletions
diff --git a/.ci/rules_python.json b/.ci/rules_python.json new file mode 100644 index 0000000..760a9ef --- /dev/null +++ b/.ci/rules_python.json @@ -0,0 +1,10 @@ +[ + { + "variation": "", + "configurations": [ + {"node": "linux-x86_64"}, + {"node": "ubuntu_16.04-x86_64"}, + {"node": "darwin-x86_64"} + ] + } +] diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..dcfa539 --- /dev/null +++ b/.gitignore @@ -0,0 +1,39 @@ +# Compiled Object files +*.slo +*.lo +*.o +*.obj + +# Precompiled Headers +*.gch +*.pch + +# Compiled Dynamic libraries +*.so +*.dylib +*.dll + +# Fortran module files +*.mod +*.smod + +# Compiled Static libraries +*.lai +*.la +*.a +*.lib + +# Executables +*.exe +*.out +*.app + +# Emacs garbage +*~ + +# Bazel directories +bazel-* +bazel-bin +bazel-genfiles +bazel-out +bazel-testlogs diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000..8b44fca --- /dev/null +++ b/.travis.yml @@ -0,0 +1,27 @@ +sudo: required +dist: trusty +language: + - java +jdk: + - oraclejdk8 # Building Bazel requires JDK8. +addons: + apt: + sources: + - sourceline: 'deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8' + key_url: 'https://storage.googleapis.com/bazel-apt/doc/apt-key.pub.gpg' + packages: + - bazel + +install: + - go get -u github.com/bazelbuild/buildifier/buildifier + +script: + # Check that all of our tools and samples build + - bazel clean && bazel build //... + # Check that all of our tests pass + - bazel clean && bazel test //... + + # Check for issues with the format of our bazel config files. + - buildifier -mode=check $(find . -name BUILD -type f) + - buildifier -mode=check $(find . -name WORKSPACE -type f) + - buildifier -mode=check $(find . -name '*.bzl' -type f) @@ -4,14 +4,17 @@ ## Rules -* [py_library](#py_library) -* [py_binary](#py_binary) +* [pip_import](docs/python/pip.md#pip_import) +* [py_library](docs/python/python.md#py_library) +* [py_binary](docs/python/python.md#py_binary) +* [py_test](docs/python/python.md#py_test) ## Overview -This is a placeholder repository that provides aliases for the native Bazel -python rules. In the future, this will also become the home for rules that -download `pip` packages, and other non-Core Python functionality. +This repository provides Python rules for Bazel. Currently, support for +rules that are available from Bazel core are simple aliases to that bundled +functionality. On top of that, this repository provides support for installing +dependencies typically managed via `pip`. ## Setup @@ -23,6 +26,11 @@ git_repository( remote = "https://github.com/bazelbuild/rules_python.git", commit = "{HEAD}", ) + +# Only needed for PIP support: +load("//python:pip.bzl", "pip_repositories") + +pip_repositories() ``` Then in your `BUILD` files load the python rules with: @@ -30,7 +38,7 @@ Then in your `BUILD` files load the python rules with: ``` python load( "@io_bazel_rules_python//python:python.bzl", - "py_binary", "py_library" + "py_binary", "py_library", "py_test", ) py_binary( @@ -39,12 +47,76 @@ py_binary( ) ``` -<a name="py_library"></a> -## py_library +## Importing `pip` dependencies + +These rules are designed to have developers continue using `requirements.txt` +to express their dependencies in a Python idiomatic manner. These dependencies +are imported into the Bazel dependency graph via a two-phased process in +`WORKSPACE`: + +```python +load("@io_bazel_rules_python//python:pip.bzl", "pip_import") + +# This rule translates the specified requirements.txt into +# @my_deps//:requirements.bzl, which itself exposes a pip_install method. +pip_import( + name = "my_deps", + requirements = "//path/to:requirements.txt", +) + +# Load the pip_install symbol for my_deps, and create the dependencies' +# repositories. +load("@my_deps//:requirements.bzl", "pip_install") +pip_install() +``` + +## Consuming `pip` dependencies + +Once a set of dependencies has been imported via `pip_import` and `pip_install` +we can start consuming them in our `py_{binary,library,test}` rules. In support +of this, the generated `requirements.bzl` also contains a `package` method, +which can be used directly in `deps=[]` to reference an imported `py_library`. + +```python +load("@my_deps//:requirements.bzl", "package") + +py_library( + name = "mylib", + srcs = ["mylib.py"], + deps = [ + ":myotherlib", + # This takes the name as specified in requirements.txt + package("importeddep"), + ] +) +``` + +## Canonical `whl_library` naming -See Bazel core [documentation](https://docs.bazel.build/versions/master/be/python.html#py_library). +It is notable that `whl_library` rules imported via `pip_import` are canonically +named, following the pattern: `pypi__{distribution}_{version}`. Characters in +these components that are illegal in Bazel label names (e.g. `-`, `.`) are +replaced with `_`. -<a name="py_binary"></a> -## py_binary +This canonical naming helps avoid redundant work to import the same library +multiple times. It is expected that this naming will remain stable, so folks +should be able to reliably depend directly on e.g. `@pypi__futures_3_1_1//:pkg` +for dependencies, however, it is recommended that folks stick with the `package` +pattern in case the need arises for us to make changes to this format in the +future. -See Bazel core [documentation](https://docs.bazel.build/versions/master/be/python.html#py_binary). +## Updating `docs/` + +All of the content (except `BUILD`) under `docs/` is generated. To update the +documentation simply run this in the root of the repository: +```shell +./update_docs.sh +``` + +## Updating `tools/` + +All of the content (except `BUILD`) under `tools/` is generated. To update the +documentation simply run this in the root of the repository: +```shell +./update_tools.sh +``` diff --git a/WORKSPACE b/WORKSPACE new file mode 100644 index 0000000..ddab657 --- /dev/null +++ b/WORKSPACE @@ -0,0 +1,109 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +workspace(name = "io_bazel_rules_python") + +# Skydoc stuff +git_repository( + name = "io_bazel_rules_sass", + remote = "https://github.com/bazelbuild/rules_sass.git", + tag = "0.0.2", +) + +load("@io_bazel_rules_sass//sass:sass.bzl", "sass_repositories") + +sass_repositories() + +git_repository( + name = "io_bazel_skydoc", + remote = "https://github.com/bazelbuild/skydoc.git", + tag = "0.1.3", +) + +load("@io_bazel_skydoc//skylark:skylark.bzl", "skydoc_repositories") + +skydoc_repositories() + +# Requirements for building our piptool. +load("//python:pip.bzl", "pip_import") + +pip_import( + name = "piptool_deps", + requirements = "//python:requirements.txt", +) + +load( + "@piptool_deps//:requirements.bzl", + _piptool_install = "pip_install", +) + +_piptool_install() + +git_repository( + name = "subpar", + remote = "https://github.com/google/subpar", + tag = "1.0.0", +) + +# Test data for WHL tool testing. +http_file( + name = "grpc_whl", + sha256 = "c232d6d168cb582e5eba8e1c0da8d64b54b041dd5ea194895a2fe76050916561", + # From https://pypi.python.org/pypi/grpcio/1.6.0 + url = ("https://pypi.python.org/packages/c6/28/" + + "67651b4eabe616b27472c5518f9b2aa3f63beab8f62100b26f05ac428639/" + + "grpcio-1.6.0-cp27-cp27m-manylinux1_i686.whl"), +) + +http_file( + name = "futures_whl", + sha256 = "c4884a65654a7c45435063e14ae85280eb1f111d94e542396717ba9828c4337f", + # From https://pypi.python.org/pypi/futures + url = ("https://pypi.python.org/packages/a6/1c/" + + "72a18c8c7502ee1b38a604a5c5243aa8c2a64f4bba4e6631b1b8972235dd/" + + "futures-3.1.1-py2-none-any.whl"), +) + +http_file( + name = "mock_whl", + sha256 = "5ce3c71c5545b472da17b72268978914d0252980348636840bd34a00b5cc96c1", + # From https://pypi.python.org/pypi/mock + url = ("https://pypi.python.org/packages/e6/35/" + + "f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/" + + "mock-2.0.0-py2.py3-none-any.whl"), +) + +# Imports for examples +pip_import( + name = "examples_helloworld", + requirements = "//examples/helloworld:requirements.txt", +) + +load( + "@examples_helloworld//:requirements.bzl", + _helloworld_install = "pip_install", +) + +_helloworld_install() + +pip_import( + name = "examples_version", + requirements = "//examples/version:requirements.txt", +) + +load( + "@examples_version//:requirements.bzl", + _version_install = "pip_install", +) + +_version_install() diff --git a/docs/BUILD b/docs/BUILD new file mode 100644 index 0000000..b2a4da3 --- /dev/null +++ b/docs/BUILD @@ -0,0 +1,60 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +package(default_visibility = ["//visibility:public"]) + +licenses(["notice"]) # Apache 2.0 + +# To regenerate html docs, run: +# ./update_docs.sh + +load("@io_bazel_skydoc//skylark:skylark.bzl", "skylark_doc", "skylark_library") + +skylark_library( + name = "whl", + srcs = ["//python:whl.bzl"], +) + +skylark_library( + name = "pip", + srcs = ["//python:pip.bzl"], +) + +skylark_library( + name = "python", + srcs = ["//python:python.bzl"], +) + +skylark_doc( + name = "docs-md", + format = "markdown", + overview = True, + site_root = ".", + deps = [ + ":pip", + ":python", + ":whl", + ], +) + +skylark_doc( + name = "docs-html", + format = "html", + overview = True, + site_root = ".", + deps = [ + ":pip", + ":python", + ":whl", + ], +) diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..d1b1dc4 --- /dev/null +++ b/docs/index.html @@ -0,0 +1,195 @@ + + +<!-- +Documentation generated by Skydoc +--> +<!DOCTYPE html> +<html> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width initial-scale=1" /> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + + <title>Overview</title> + + <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,600,700" type="text/css"> + <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons"> + <link rel="stylesheet" href="https://code.getmdl.io/1.1.1/material.green-light_blue.min.css"> + <script defer src="https://code.getmdl.io/1.1.1/material.min.js"></script> + <link rel="stylesheet" href="./main.css"> + </head> + <body> + <div class="mdl-layout mdl-js-layout mdl-layout--fixed-drawer + mdl-layout--fixed-header"> + <header class="mdl-layout__header"> + <div class="mdl-layout__header-row"> + <span class="mdl-layout-title">Overview</span> + </div> + </header> + <div class="mdl-layout__drawer"> + <span class="mdl-layout-title">Bazel</span> + <nav class="drawer-nav"> + <ul class="drawer-nav"> + +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/python.html">python Rules</a> + <ul> + </ul> +</li> +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/pip.html">Import pip requirements into Bazel.</a> + <ul> + </ul> +</li> +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/whl.html">Import .whl files into Bazel.</a> + <ul> + </ul> +</li> + + </ul> + </nav> + </div> + + <main class="mdl-layout__content"> + <div class="page-content"> +<h1>Overview</h1> + + +<nav class="toc"> + <h2>Rule sets</h2> + <ul> + <li><a href="#python">python Rules</a></li> + <li><a href="#pip">Import pip requirements into Bazel.</a></li> + <li><a href="#whl">Import .whl files into Bazel.</a></li> + </ul> +</nav> + +<h2><a href="./python/python.html">python Rules</a></h2> + +<h3>Macros</h3> +<table class="overview-table"> + <colgroup> + <col class="col-name" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr> + <td> + <a href="./python/python.html#py_library"> + <code>py_library</code> + </a> + </td> + <td> + <p>See the Bazel core py_library documentation.</p> + + </td> + </tr> + <tr> + <td> + <a href="./python/python.html#py_binary"> + <code>py_binary</code> + </a> + </td> + <td> + <p>See the Bazel core py_binary documentation.</p> + + </td> + </tr> + <tr> + <td> + <a href="./python/python.html#py_test"> + <code>py_test</code> + </a> + </td> + <td> + <p>See the Bazel core py_test documentation.</p> + + </td> + </tr> + </tbody> +</table> +<h2><a href="./python/pip.html">Import pip requirements into Bazel.</a></h2> + +<h3>Macros</h3> +<table class="overview-table"> + <colgroup> + <col class="col-name" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr> + <td> + <a href="./python/pip.html#pip_repositories"> + <code>pip_repositories</code> + </a> + </td> + <td> + <p>Pull in dependencies needed for pulling in pip dependencies.</p> + + </td> + </tr> + </tbody> +</table> +<h3>Repository Rules</h3> +<table class="overview-table"> + <colgroup> + <col class="col-name" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr> + <td> + <a href="./python/pip.html#pip_import"> + <code>pip_import</code> + </a> + </td> + <td> + <p>A rule for importing <code>requirements.txt</code> dependencies into Bazel.</p> + + </td> + </tr> + </tbody> +</table> +<h2><a href="./python/whl.html">Import .whl files into Bazel.</a></h2> + +<h3>Repository Rules</h3> +<table class="overview-table"> + <colgroup> + <col class="col-name" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr> + <td> + <a href="./python/whl.html#whl_library"> + <code>whl_library</code> + </a> + </td> + <td> + <p>A rule for importing <code>.whl</code> dependencies into Bazel.</p> + + </td> + </tr> + </tbody> +</table> + + + </div> + + <footer class="mdl-mini-footer"> + <div class="mdl-mini-footer__left-section"> + <div class="mdl-logo">Bazel</div> + <ul class="mdl-mini-footer__link-list"> + <li><a href="http://bazel.io">Home</a></li> + <li><a href="https://github.com/bazelbuild">GitHub</a></li> + </ul> + </div> + </footer> + </main> + </div> + </body> +</html> diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..febdbcb --- /dev/null +++ b/docs/index.md @@ -0,0 +1,121 @@ + +# Overview + + +<nav class="toc"> + <h2>Rule sets</h2> + <ul> + <li><a href="#python">python Rules</a></li> + <li><a href="#pip">Import pip requirements into Bazel.</a></li> + <li><a href="#whl">Import .whl files into Bazel.</a></li> + </ul> +</nav> + +<h2><a href="./python/python.html">python Rules</a></h2> + +<h3>Macros</h3> +<table class="overview-table"> + <colgroup> + <col class="col-name" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr> + <td> + <a href="./python/python.html#py_library"> + <code>py_library</code> + </a> + </td> + <td> + <p>See the Bazel core py_library documentation.</p> + + </td> + </tr> + <tr> + <td> + <a href="./python/python.html#py_binary"> + <code>py_binary</code> + </a> + </td> + <td> + <p>See the Bazel core py_binary documentation.</p> + + </td> + </tr> + <tr> + <td> + <a href="./python/python.html#py_test"> + <code>py_test</code> + </a> + </td> + <td> + <p>See the Bazel core py_test documentation.</p> + + </td> + </tr> + </tbody> +</table> +<h2><a href="./python/pip.html">Import pip requirements into Bazel.</a></h2> + +<h3>Macros</h3> +<table class="overview-table"> + <colgroup> + <col class="col-name" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr> + <td> + <a href="./python/pip.html#pip_repositories"> + <code>pip_repositories</code> + </a> + </td> + <td> + <p>Pull in dependencies needed for pulling in pip dependencies.</p> + + </td> + </tr> + </tbody> +</table> +<h3>Repository Rules</h3> +<table class="overview-table"> + <colgroup> + <col class="col-name" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr> + <td> + <a href="./python/pip.html#pip_import"> + <code>pip_import</code> + </a> + </td> + <td> + <p>A rule for importing <code>requirements.txt</code> dependencies into Bazel.</p> + + </td> + </tr> + </tbody> +</table> +<h2><a href="./python/whl.html">Import .whl files into Bazel.</a></h2> + +<h3>Repository Rules</h3> +<table class="overview-table"> + <colgroup> + <col class="col-name" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr> + <td> + <a href="./python/whl.html#whl_library"> + <code>whl_library</code> + </a> + </td> + <td> + <p>A rule for importing <code>.whl</code> dependencies into Bazel.</p> + + </td> + </tr> + </tbody> +</table> diff --git a/docs/main.css b/docs/main.css new file mode 100755 index 0000000..f506e12 --- /dev/null +++ b/docs/main.css @@ -0,0 +1,3 @@ +body{background-color:#fafafa}pre,code{font-family:'Liberation Mono', Consolas, Monaco, 'Andale Mono', monospace}pre{background-color:#eee;padding:20px;overflow-x:auto;word-wrap:normal}pre code{overflow-wrap:normal;white-space:pre}code{display:inline-block;font-size:90%;white-space:pre-wrap}.mdl-layout__drawer{background-color:#fff}.mdl-layout__drawer .mdl-layout-title{border-bottom:1px solid #e0e0e0;padding-left:24px}.drawer-nav ul{list-style:none;padding-left:0}.drawer-nav ul li{display:block;padding:0}.drawer-nav ul li ul li a{padding-left:44px;font-weight:400}.drawer-nav ul li a{display:block;flex-shrink:0;padding:15px 0 15px 22px;margin:0;font-weight:600;color:#757575;line-height:1em;text-decoration:none;cursor:pointer}.drawer-nav ul li a:active,.drawer-nav ul li a:hover{background-color:#f0f0f0}.drawer-nav ul li.active a{color:#4caf50;font-weight:500}h1.page-title{font-size:34px;font-weight:400;line-height:40px;margin-bottom:30px;color:#4caf50}p.lead{font-size:20px;line-height:32px}table{border-collapse:collapse;border-spacing:0;background-color:#fff;table-layout:auto}table thead th{background-color:#fafafa;border:1px solid #eee;color:#757575;padding:12px 12px 12px 24px;vertical-align:top}table tbody td{border:1px solid #eee;padding:12px 12px 12px 24px;vertical-align:top}table.params-table{width:100%}table.params-table col.col-param{width:25%}table.params-table col.col-description{width:75%}table.overview-table{width:100%}table.overview-table col.col-name{width:25%}table.overview-table col.col-description{width:75%}table.overview-table td p{margin:0}hr{margin-top:80px;margin-bottom:80px}nav.toc{border-left:5px solid #4caf50;padding-left:20px;margin-bottom:48px}nav.toc h1,nav.toc h2{font-size:15px;line-height:16px;padding-bottom:12px;margin-bottom:0;font-weight:400;color:#757575}nav.toc ul{list-style:none;margin-top:0;padding-left:0}nav.toc ul li{font-size:20px;line-height:40px}nav.toc ul li a{color:#4caf50}.page-content{margin-left:auto;margin-right:auto;padding-top:60px;padding-bottom:60px;width:760px}.page-content a{text-decoration:none}.page-content h1{font-size:34px;font-weight:400;line-height:40px;margin-bottom:30px;color:#4caf50}.page-content h2{font-size:24px;font-weight:400;line-height:32px;margin-bottom:30px;color:#4caf50}.page-content h3{font-size:20px;font-weight:400;line-height:32px;margin-bottom:30px;color:#4caf50}@media (max-width: 768px){.page-content{width:360px}}@media (min-width: 768px){.page-content{width:760px}}@media (min-width: 1476px){.page-content{width:1160px}}.mdl-mini-footer{padding-left:40px} + +/*# sourceMappingURL=main.css.map */
\ No newline at end of file diff --git a/docs/python/pip.html b/docs/python/pip.html new file mode 100644 index 0000000..2d43cb7 --- /dev/null +++ b/docs/python/pip.html @@ -0,0 +1,161 @@ + + +<!-- +Documentation generated by Skydoc +--> +<!DOCTYPE html> +<html> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width initial-scale=1" /> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + + <title>Import pip requirements into Bazel.</title> + + <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,600,700" type="text/css"> + <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons"> + <link rel="stylesheet" href="https://code.getmdl.io/1.1.1/material.green-light_blue.min.css"> + <script defer src="https://code.getmdl.io/1.1.1/material.min.js"></script> + <link rel="stylesheet" href="./main.css"> + </head> + <body> + <div class="mdl-layout mdl-js-layout mdl-layout--fixed-drawer + mdl-layout--fixed-header"> + <header class="mdl-layout__header"> + <div class="mdl-layout__header-row"> + <span class="mdl-layout-title">Import pip requirements into Bazel.</span> + </div> + </header> + <div class="mdl-layout__drawer"> + <span class="mdl-layout-title">Bazel</span> + <nav class="drawer-nav"> + <ul class="drawer-nav"> + +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/python.html">python Rules</a> + <ul> + </ul> +</li> +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/pip.html">Import pip requirements into Bazel.</a> + <ul> + </ul> +</li> +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/whl.html">Import .whl files into Bazel.</a> + <ul> + </ul> +</li> + + </ul> + </nav> + </div> + + <main class="mdl-layout__content"> + <div class="page-content"> + <h1>Import pip requirements into Bazel.</h1> + +<nav class="toc"> + <h2>Repository Rules</h2> + <ul> + <li><a href="#pip_import">pip_import</a></li> + </ul> + <h2>Macros</h2> + <ul> + <li><a href="#pip_repositories">pip_repositories</a></li> + </ul> +</nav> + <hr> + + <h2 id="pip_repositories">pip_repositories</h2> + + <pre>pip_repositories()</pre> + + <p>Pull in dependencies needed for pulling in pip dependencies.</p> +<p>A placeholder method that will eventually pull in any dependencies +needed to install pip dependencies.</p> + + + <hr> + + <h2 id="pip_import">pip_import</h2> + + <pre>pip_import(<a href="#pip_import.name">name</a>, <a href="#pip_import.requirements">requirements</a>)</pre> + + <p>A rule for importing <code>requirements.txt</code> dependencies into Bazel.</p> +<p>This rule imports a <code>requirements.txt</code> file and generates a new +<code>requirements.bzl</code> file. This is used via the <code>WORKSPACE</code> +pattern:</p> +<pre><code>pip_import( + name = "foo", + requirements = ":requirements.txt", +) +load("@foo//:requirements.bzl", "pip_install") +pip_install() +</code></pre><p>You can then reference imported dependencies from your <code>BUILD</code> +file with:</p> +<pre><code>load("@foo//:requirements.bzl", "package") +py_library( + name = "bar", + ... + deps = [ + "//my/other:dep", + package("futures"), + package("mock"), + ], +) +</code></pre><p>Or alternatively:</p> +<pre><code>load("@foo//:requirements.bzl", "all_packages") +py_binary( + name = "baz", + ... + deps = [ + ":foo", + ] + all_packages, +) +</code></pre> + + <h3 id="pip_import_args">Attributes</h3> + +<table class="params-table"> + <colgroup> + <col class="col-param" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr id="pip_import.name"> + <td><code>name</code></td> + <td> + <p><code><a href="https://bazel.build/docs/build-ref.html#name">Name</a>; Required</code></p> + <p>A unique name for this rule.</p> + </td> + </tr> + <tr id="pip_import.requirements"> + <td><code>requirements</code></td> + <td> + <p><code><a href="https://bazel.build/docs/build-ref.html#labels">Label</a>; Required</code></p> + <p>The label of a requirements.txt file.</p> + </td> + </tr> + </tbody> +</table> + + + </div> + + <footer class="mdl-mini-footer"> + <div class="mdl-mini-footer__left-section"> + <div class="mdl-logo">Bazel</div> + <ul class="mdl-mini-footer__link-list"> + <li><a href="http://bazel.io">Home</a></li> + <li><a href="https://github.com/bazelbuild">GitHub</a></li> + </ul> + </div> + </footer> + </main> + </div> + </body> +</html> diff --git a/docs/python/pip.md b/docs/python/pip.md new file mode 100644 index 0000000..2a373d1 --- /dev/null +++ b/docs/python/pip.md @@ -0,0 +1,101 @@ + +<!--- +Documentation generated by Skydoc +--> +<h1>Import pip requirements into Bazel.</h1> + + +<nav class="toc"> + <h2>Repository Rules</h2> + <ul> + <li><a href="#pip_import">pip_import</a></li> + </ul> + <h2>Macros</h2> + <ul> + <li><a href="#pip_repositories">pip_repositories</a></li> + </ul> +</nav> +<a name="pip_repositories"></a> +## pip_repositories + +<pre> +pip_repositories() +</pre> + +Pull in dependencies needed for pulling in pip dependencies. + +A placeholder method that will eventually pull in any dependencies +needed to install pip dependencies. + +<a name="pip_import"></a> +## pip_import + +<pre> +pip_import(<a href="#pip_import.name">name</a>, <a href="#pip_import.requirements">requirements</a>) +</pre> + +A rule for importing <code>requirements.txt</code> dependencies into Bazel. + +This rule imports a <code>requirements.txt</code> file and generates a new +<code>requirements.bzl</code> file. This is used via the <code>WORKSPACE</code> +pattern: +<pre><code>pip_import( + name = "foo", + requirements = ":requirements.txt", +) +load("@foo//:requirements.bzl", "pip_install") +pip_install() +</code></pre> + +You can then reference imported dependencies from your <code>BUILD</code> +file with: +<pre><code>load("@foo//:requirements.bzl", "package") +py_library( + name = "bar", + ... + deps = [ + "//my/other:dep", + package("futures"), + package("mock"), + ], +) +</code></pre> + +Or alternatively: +<pre><code>load("@foo//:requirements.bzl", "all_packages") +py_binary( + name = "baz", + ... + deps = [ + ":foo", + ] + all_packages, +) +</code></pre> + + +<a name="pip_import_args"></a> +### Attributes + + +<table class="params-table"> + <colgroup> + <col class="col-param" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr id="pip_import.name"> + <td><code>name</code></td> + <td> + <p><code><a href="https://bazel.build/docs/build-ref.html#name">Name</a>; Required</code></p> + <p>A unique name for this rule.</p> + </td> + </tr> + <tr id="pip_import.requirements"> + <td><code>requirements</code></td> + <td> + <p><code><a href="https://bazel.build/docs/build-ref.html#labels">Label</a>; Required</code></p> + <p>The label of a requirements.txt file.</p> + </td> + </tr> + </tbody> +</table> diff --git a/docs/python/python.html b/docs/python/python.html new file mode 100644 index 0000000..c98039a --- /dev/null +++ b/docs/python/python.html @@ -0,0 +1,115 @@ + + +<!-- +Documentation generated by Skydoc +--> +<!DOCTYPE html> +<html> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width initial-scale=1" /> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + + <title>python Rules</title> + + <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,600,700" type="text/css"> + <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons"> + <link rel="stylesheet" href="https://code.getmdl.io/1.1.1/material.green-light_blue.min.css"> + <script defer src="https://code.getmdl.io/1.1.1/material.min.js"></script> + <link rel="stylesheet" href="./main.css"> + </head> + <body> + <div class="mdl-layout mdl-js-layout mdl-layout--fixed-drawer + mdl-layout--fixed-header"> + <header class="mdl-layout__header"> + <div class="mdl-layout__header-row"> + <span class="mdl-layout-title">python Rules</span> + </div> + </header> + <div class="mdl-layout__drawer"> + <span class="mdl-layout-title">Bazel</span> + <nav class="drawer-nav"> + <ul class="drawer-nav"> + +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/python.html">python Rules</a> + <ul> + </ul> +</li> +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/pip.html">Import pip requirements into Bazel.</a> + <ul> + </ul> +</li> +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/whl.html">Import .whl files into Bazel.</a> + <ul> + </ul> +</li> + + </ul> + </nav> + </div> + + <main class="mdl-layout__content"> + <div class="page-content"> + <h1>python Rules</h1> + +<nav class="toc"> + <h2>Macros</h2> + <ul> + <li><a href="#py_library">py_library</a></li> + <li><a href="#py_binary">py_binary</a></li> + <li><a href="#py_test">py_test</a></li> + </ul> +</nav> + <hr> + + <h2 id="py_library">py_library</h2> + + <pre>py_library()</pre> + + <p>See the Bazel core py_library documentation.</p> +<p><a href="https://docs.bazel.build/versions/master/be/python.html#py_library">available here</a>.</p> + + + <hr> + + <h2 id="py_binary">py_binary</h2> + + <pre>py_binary()</pre> + + <p>See the Bazel core py_binary documentation.</p> +<p><a href="https://docs.bazel.build/versions/master/be/python.html#py_binary">available here</a>.</p> + + + <hr> + + <h2 id="py_test">py_test</h2> + + <pre>py_test()</pre> + + <p>See the Bazel core py_test documentation.</p> +<p><a href="https://docs.bazel.build/versions/master/be/python.html#py_test">available here</a>.</p> + + + + + </div> + + <footer class="mdl-mini-footer"> + <div class="mdl-mini-footer__left-section"> + <div class="mdl-logo">Bazel</div> + <ul class="mdl-mini-footer__link-list"> + <li><a href="http://bazel.io">Home</a></li> + <li><a href="https://github.com/bazelbuild">GitHub</a></li> + </ul> + </div> + </footer> + </main> + </div> + </body> +</html> diff --git a/docs/python/python.md b/docs/python/python.md new file mode 100644 index 0000000..a70c86a --- /dev/null +++ b/docs/python/python.md @@ -0,0 +1,51 @@ + +<!--- +Documentation generated by Skydoc +--> +<h1>python Rules</h1> + + +<nav class="toc"> + <h2>Macros</h2> + <ul> + <li><a href="#py_library">py_library</a></li> + <li><a href="#py_binary">py_binary</a></li> + <li><a href="#py_test">py_test</a></li> + </ul> +</nav> +<a name="py_library"></a> +## py_library + +<pre> +py_library() +</pre> + +See the Bazel core py_library documentation. + +[available here]( +https://docs.bazel.build/versions/master/be/python.html#py_library). + +<a name="py_binary"></a> +## py_binary + +<pre> +py_binary() +</pre> + +See the Bazel core py_binary documentation. + +[available here]( +https://docs.bazel.build/versions/master/be/python.html#py_binary). + +<a name="py_test"></a> +## py_test + +<pre> +py_test() +</pre> + +See the Bazel core py_test documentation. + +[available here]( +https://docs.bazel.build/versions/master/be/python.html#py_test). + diff --git a/docs/python/whl.html b/docs/python/whl.html new file mode 100644 index 0000000..5fcd672 --- /dev/null +++ b/docs/python/whl.html @@ -0,0 +1,135 @@ + + +<!-- +Documentation generated by Skydoc +--> +<!DOCTYPE html> +<html> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width initial-scale=1" /> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + + <title>Import .whl files into Bazel.</title> + + <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,600,700" type="text/css"> + <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons"> + <link rel="stylesheet" href="https://code.getmdl.io/1.1.1/material.green-light_blue.min.css"> + <script defer src="https://code.getmdl.io/1.1.1/material.min.js"></script> + <link rel="stylesheet" href="./main.css"> + </head> + <body> + <div class="mdl-layout mdl-js-layout mdl-layout--fixed-drawer + mdl-layout--fixed-header"> + <header class="mdl-layout__header"> + <div class="mdl-layout__header-row"> + <span class="mdl-layout-title">Import .whl files into Bazel.</span> + </div> + </header> + <div class="mdl-layout__drawer"> + <span class="mdl-layout-title">Bazel</span> + <nav class="drawer-nav"> + <ul class="drawer-nav"> + +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/python.html">python Rules</a> + <ul> + </ul> +</li> +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/pip.html">Import pip requirements into Bazel.</a> + <ul> + </ul> +</li> +<li><a href="./index.html">Overview</a></li> +<li> + <a href="./python/whl.html">Import .whl files into Bazel.</a> + <ul> + </ul> +</li> + + </ul> + </nav> + </div> + + <main class="mdl-layout__content"> + <div class="page-content"> + <h1>Import .whl files into Bazel.</h1> + +<nav class="toc"> + <h2>Repository Rules</h2> + <ul> + <li><a href="#whl_library">whl_library</a></li> + </ul> +</nav> + <hr> + + <h2 id="whl_library">whl_library</h2> + + <pre>whl_library(<a href="#whl_library.name">name</a>, <a href="#whl_library.requirements">requirements</a>, <a href="#whl_library.whl">whl</a>)</pre> + + <p>A rule for importing <code>.whl</code> dependencies into Bazel.</p> +<p><b>This rule is currently used to implement <code>pip_import</code>, +it is not intended to work standalone, and the interface may change.</b> +See <code>pip_import</code> for proper usage.</p> +<p>This rule imports a <code>.whl</code> file as a <code>py_library</code>:</p> +<pre><code>whl_library( + name = "foo", + whl = ":my-whl-file", + requirements = "name of pip_import rule", +) +</code></pre><p>This rule defines a <code>@foo//:pkg</code> <code>py_library</code> target.</p> + + + <h3 id="whl_library_args">Attributes</h3> + +<table class="params-table"> + <colgroup> + <col class="col-param" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr id="whl_library.name"> + <td><code>name</code></td> + <td> + <p><code><a href="https://bazel.build/docs/build-ref.html#name">Name</a>; Required</code></p> + <p>A unique name for this rule.</p> + </td> + </tr> + <tr id="whl_library.requirements"> + <td><code>requirements</code></td> + <td> + <p><code>String; Optional; Default is ''</code></p> + <p>The name of the pip_import repository rule from which to +load this .whl's dependencies.</p> + </td> + </tr> + <tr id="whl_library.whl"> + <td><code>whl</code></td> + <td> + <p><code><a href="https://bazel.build/docs/build-ref.html#labels">Label</a>; Required</code></p> + <p>The path to the .whl file (the name is expected to follow <a href="https://www.python.org/dev/peps/pep-0427/#file-name-convention">this +convention</a>)</p> + </td> + </tr> + </tbody> +</table> + + + </div> + + <footer class="mdl-mini-footer"> + <div class="mdl-mini-footer__left-section"> + <div class="mdl-logo">Bazel</div> + <ul class="mdl-mini-footer__link-list"> + <li><a href="http://bazel.io">Home</a></li> + <li><a href="https://github.com/bazelbuild">GitHub</a></li> + </ul> + </div> + </footer> + </main> + </div> + </body> +</html> diff --git a/docs/python/whl.md b/docs/python/whl.md new file mode 100644 index 0000000..41467ce --- /dev/null +++ b/docs/python/whl.md @@ -0,0 +1,72 @@ + +<!--- +Documentation generated by Skydoc +--> +<h1>Import .whl files into Bazel.</h1> + + +<nav class="toc"> + <h2>Repository Rules</h2> + <ul> + <li><a href="#whl_library">whl_library</a></li> + </ul> +</nav> +<a name="whl_library"></a> +## whl_library + +<pre> +whl_library(<a href="#whl_library.name">name</a>, <a href="#whl_library.requirements">requirements</a>, <a href="#whl_library.whl">whl</a>) +</pre> + +A rule for importing <code>.whl</code> dependencies into Bazel. + +<b>This rule is currently used to implement <code>pip_import</code>, +it is not intended to work standalone, and the interface may change.</b> +See <code>pip_import</code> for proper usage. + +This rule imports a <code>.whl</code> file as a <code>py_library</code>: +<pre><code>whl_library( + name = "foo", + whl = ":my-whl-file", + requirements = "name of pip_import rule", +) +</code></pre> + +This rule defines a <code>@foo//:pkg</code> <code>py_library</code> target. + + +<a name="whl_library_args"></a> +### Attributes + + +<table class="params-table"> + <colgroup> + <col class="col-param" /> + <col class="col-description" /> + </colgroup> + <tbody> + <tr id="whl_library.name"> + <td><code>name</code></td> + <td> + <p><code><a href="https://bazel.build/docs/build-ref.html#name">Name</a>; Required</code></p> + <p>A unique name for this rule.</p> + </td> + </tr> + <tr id="whl_library.requirements"> + <td><code>requirements</code></td> + <td> + <p><code>String; Optional; Default is ''</code></p> + <p>The name of the pip_import repository rule from which to +load this .whl's dependencies.</p> + </td> + </tr> + <tr id="whl_library.whl"> + <td><code>whl</code></td> + <td> + <p><code><a href="https://bazel.build/docs/build-ref.html#labels">Label</a>; Required</code></p> + <p>The path to the .whl file (the name is expected to follow <a href="https://www.python.org/dev/peps/pep-0427/#file-name-convention">this +convention</a>)</p> + </td> + </tr> + </tbody> +</table> diff --git a/examples/BUILD b/examples/BUILD new file mode 100644 index 0000000..426ddc0 --- /dev/null +++ b/examples/BUILD @@ -0,0 +1,16 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +package(default_visibility = ["//visibility:public"]) + +licenses(["notice"]) # Apache 2.0 diff --git a/examples/helloworld/BUILD b/examples/helloworld/BUILD new file mode 100644 index 0000000..ccb8e36 --- /dev/null +++ b/examples/helloworld/BUILD @@ -0,0 +1,31 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +package(default_visibility = ["//visibility:public"]) + +licenses(["notice"]) # Apache 2.0 + +load("@examples_helloworld//:requirements.bzl", "package") +load("//python:python.bzl", "py_library", "py_test") + +py_library( + name = "helloworld", + srcs = ["helloworld.py"], + deps = [package("futures")], +) + +py_test( + name = "helloworld_test", + srcs = ["helloworld_test.py"], + deps = [":helloworld"], +) diff --git a/examples/helloworld/helloworld.py b/examples/helloworld/helloworld.py new file mode 100644 index 0000000..b629e80 --- /dev/null +++ b/examples/helloworld/helloworld.py @@ -0,0 +1,29 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from concurrent import futures + + +class HelloWorld(object): + def __init__(self): + self._threadpool = futures.ThreadPoolExecutor(max_workers=5) + + def SayHello(self): + print("Hello World") + + def SayHelloAsync(self): + self._threadpool.submit(self.SayHello) + + def Stop(self): + self._threadpool.shutdown(wait = True) diff --git a/examples/helloworld/helloworld_test.py b/examples/helloworld/helloworld_test.py new file mode 100644 index 0000000..ca9f6b1 --- /dev/null +++ b/examples/helloworld/helloworld_test.py @@ -0,0 +1,41 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import unittest + +from examples.helloworld import helloworld + + +class HelloWorldTest(unittest.TestCase): + + def test_helloworld(self): + hw = helloworld.HelloWorld() + hw.SayHello() + + def test_helloworld_async(self): + hw = helloworld.HelloWorld() + hw.SayHelloAsync() + hw.Stop() + + def test_helloworld_multiple(self): + hw = helloworld.HelloWorld() + hw.SayHelloAsync() + hw.SayHelloAsync() + hw.SayHelloAsync() + hw.SayHelloAsync() + hw.Stop() + + +if __name__ == '__main__': + unittest.main() diff --git a/examples/helloworld/requirements.txt b/examples/helloworld/requirements.txt new file mode 100644 index 0000000..372420d --- /dev/null +++ b/examples/helloworld/requirements.txt @@ -0,0 +1 @@ +futures>=3.1 diff --git a/examples/version/BUILD b/examples/version/BUILD new file mode 100644 index 0000000..e999ba6 --- /dev/null +++ b/examples/version/BUILD @@ -0,0 +1,25 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +package(default_visibility = ["//visibility:public"]) + +licenses(["notice"]) # Apache 2.0 + +load("@examples_version//:requirements.bzl", "package") +load("//python:python.bzl", "py_test") + +py_test( + name = "version_test", + srcs = ["version_test.py"], + deps = [package("pip")], +) diff --git a/examples/version/requirements.txt b/examples/version/requirements.txt new file mode 100644 index 0000000..433e330 --- /dev/null +++ b/examples/version/requirements.txt @@ -0,0 +1 @@ +pip==9.0.1 diff --git a/examples/version/version_test.py b/examples/version/version_test.py new file mode 100644 index 0000000..9b469b7 --- /dev/null +++ b/examples/version/version_test.py @@ -0,0 +1,26 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pip +import unittest + + +class VersionTest(unittest.TestCase): + + def test_version(self): + self.assertEqual(pip.__version__, '9.0.1') + + +if __name__ == '__main__': + unittest.main() diff --git a/python/BUILD b/python/BUILD index 426ddc0..be4b731 100644 --- a/python/BUILD +++ b/python/BUILD @@ -14,3 +14,11 @@ package(default_visibility = ["//visibility:public"]) licenses(["notice"]) # Apache 2.0 + +exports_files([ + "pip.bzl", + "pip.sh", + "python.bzl", + "whl.bzl", + "whl.sh", +]) diff --git a/python/pip.bzl b/python/pip.bzl new file mode 100644 index 0000000..0345b50 --- /dev/null +++ b/python/pip.bzl @@ -0,0 +1,101 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Import pip requirements into Bazel.""" + +def _pip_import_impl(repository_ctx): + """Core implementation of pip_import.""" + + # Add an empty top-level BUILD file. + # This is because Bazel requires BUILD files along all paths accessed + # via //this/sort/of:path and we wouldn't be able to load our generated + # requirements.bzl without it. + repository_ctx.file("BUILD", "") + + # To see the output, pass: quiet=False + result = repository_ctx.execute([ + "python", repository_ctx.path(repository_ctx.attr._script), + "--name", repository_ctx.attr.name, + "--input", repository_ctx.path(repository_ctx.attr.requirements), + "--output", repository_ctx.path("requirements.bzl"), + "--directory", repository_ctx.path(""), + ]) + + if result.return_code: + fail("pip_import failed: %s (%s)" % (result.stdout, result.stderr)) + +pip_import = repository_rule( + attrs = { + "requirements": attr.label( + allow_files = True, + mandatory = True, + single_file = True, + ), + "_script": attr.label( + executable = True, + default = Label("//tools:piptool.par"), + cfg = "host", + ), + }, + implementation = _pip_import_impl, +) + +"""A rule for importing <code>requirements.txt</code> dependencies into Bazel. + +This rule imports a <code>requirements.txt</code> file and generates a new +<code>requirements.bzl</code> file. This is used via the <code>WORKSPACE</code> +pattern: +<pre><code>pip_import( + name = "foo", + requirements = ":requirements.txt", +) +load("@foo//:requirements.bzl", "pip_install") +pip_install() +</code></pre> + +You can then reference imported dependencies from your <code>BUILD</code> +file with: +<pre><code>load("@foo//:requirements.bzl", "package") +py_library( + name = "bar", + ... + deps = [ + "//my/other:dep", + package("futures"), + package("mock"), + ], +) +</code></pre> + +Or alternatively: +<pre><code>load("@foo//:requirements.bzl", "all_packages") +py_binary( + name = "baz", + ... + deps = [ + ":foo", + ] + all_packages, +) +</code></pre> + +Args: + requirements: The label of a requirements.txt file. +""" + +def pip_repositories(): + """Pull in dependencies needed for pulling in pip dependencies. + + A placeholder method that will eventually pull in any dependencies + needed to install pip dependencies. + """ + pass diff --git a/python/python.bzl b/python/python.bzl index fca849d..8cbb11a 100644 --- a/python/python.bzl +++ b/python/python.bzl @@ -13,7 +13,25 @@ # limitations under the License. def py_library(*args, **kwargs): + """See the Bazel core py_library documentation. + + [available here]( + https://docs.bazel.build/versions/master/be/python.html#py_library). + """ native.py_library(*args, **kwargs) def py_binary(*args, **kwargs): + """See the Bazel core py_binary documentation. + + [available here]( + https://docs.bazel.build/versions/master/be/python.html#py_binary). + """ native.py_binary(*args, **kwargs) + +def py_test(*args, **kwargs): + """See the Bazel core py_test documentation. + + [available here]( + https://docs.bazel.build/versions/master/be/python.html#py_test). + """ + native.py_test(*args, **kwargs) diff --git a/python/requirements.txt b/python/requirements.txt new file mode 100644 index 0000000..0fd2ec6 --- /dev/null +++ b/python/requirements.txt @@ -0,0 +1,2 @@ +pip==9.0.1 +wheel==0.30.0a0 diff --git a/python/whl.bzl b/python/whl.bzl new file mode 100644 index 0000000..f7827ed --- /dev/null +++ b/python/whl.bzl @@ -0,0 +1,67 @@ +# Copyright 2017 Google Inc. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Import .whl files into Bazel.""" + +def _whl_impl(repository_ctx): + """Core implementation of whl_library.""" + + result = repository_ctx.execute([ + "python", + repository_ctx.path(repository_ctx.attr._script), + "--whl", repository_ctx.path(repository_ctx.attr.whl), + "--requirements", repository_ctx.attr.requirements, + ]) + if result.return_code: + fail("whl_library failed: %s (%s)" % (result.stdout, result.stderr)) + +whl_library = repository_rule( + attrs = { + "whl": attr.label( + allow_files = True, + mandatory = True, + single_file = True, + ), + "requirements": attr.string(), + "_script": attr.label( + executable = True, + default = Label("//rules_python:whl.py"), + cfg = "host", + ), + }, + implementation = _whl_impl, +) + +"""A rule for importing <code>.whl</code> dependencies into Bazel. + +<b>This rule is currently used to implement <code>pip_import</code>, +it is not intended to work standalone, and the interface may change.</b> +See <code>pip_import</code> for proper usage. + +This rule imports a <code>.whl</code> file as a <code>py_library</code>: +<pre><code>whl_library( + name = "foo", + whl = ":my-whl-file", + requirements = "name of pip_import rule", +) +</code></pre> + +This rule defines a <code>@foo//:pkg</code> <code>py_library</code> target. + +Args: + whl: The path to the .whl file (the name is expected to follow [this + convention](https://www.python.org/dev/peps/pep-0427/#file-name-convention)) + + requirements: The name of the pip_import repository rule from which to + load this .whl's dependencies. +""" diff --git a/rules_python/BUILD b/rules_python/BUILD new file mode 100644 index 0000000..a0534fd --- /dev/null +++ b/rules_python/BUILD @@ -0,0 +1,47 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +package(default_visibility = ["//visibility:public"]) + +licenses(["notice"]) # Apache 2.0 + +load("//python:python.bzl", "py_binary", "py_library", "py_test") + +py_library( + name = "whl", + srcs = ["whl.py"], +) + +py_test( + name = "whl_test", + srcs = ["whl_test.py"], + data = [ + "@futures_whl//file", + "@grpc_whl//file", + "@mock_whl//file", + ], + deps = [":whl"], +) + +load("@subpar//:subpar.bzl", "par_binary") +load("@piptool_deps//:requirements.bzl", "all_packages") + +# TODO(mattmoor): Bundle this tool as a PAR without any +# system-installed pre-requisites. See TODOs in piptool.py. +par_binary( + name = "piptool", + srcs = ["piptool.py"], + deps = [ + ":whl", + ] + all_packages, +) diff --git a/rules_python/piptool.py b/rules_python/piptool.py new file mode 100644 index 0000000..b5f4121 --- /dev/null +++ b/rules_python/piptool.py @@ -0,0 +1,154 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""The piptool module imports pip requirements into Bazel rules.""" + +import argparse +import json +import os +import pkgutil +import re +import sys +import tempfile +import zipfile + +# TODO(mattmoor): When this tool is invoked bundled as a PAR file, +# but not as a py_binary, we get a warning that indicates the system +# installed version of PIP is being picked up instead of our bundled +# version, which should be 9.0.1, e.g. +# You are using pip version 1.5.4, however version 9.0.1 is available. +# You should consider upgrading via the 'pip install --upgrade pip' command. +try: + # Make sure we're using a suitable version of pip as a library. + # Fallback on using it as a CLI. + from pip._vendor import requests + + from pip import main as _pip_main + def pip_main(argv): + # Extract the certificates from the PAR following the example of get-pip.py + # https://github.com/pypa/get-pip/blob/430ba37776ae2ad89/template.py#L164-L168 + cert_path = os.path.join(tempfile.mkdtemp(), "cacert.pem") + with open(cert_path, "wb") as cert: + cert.write(pkgutil.get_data("pip._vendor.requests", "cacert.pem")) + return _pip_main(argv + ["--cert", cert_path]) + +except: + import subprocess + + def pip_main(argv): + return subprocess.call(['pip'] + argv) + +# TODO(mattmoor): We can't easily depend on other libraries when +# being invoked as a raw .py file. Once bundled, we should be able +# to remove this fallback on a stub implementation of Wheel. +try: + from rules_python.whl import Wheel +except: + class Wheel(object): + + def __init__(self, path): + self._path = path + + def basename(self): + return os.path.basename(self._path) + + def distribution(self): + # See https://www.python.org/dev/peps/pep-0427/#file-name-convention + parts = self.basename().split('-') + return parts[0] + + def version(self): + # See https://www.python.org/dev/peps/pep-0427/#file-name-convention + parts = self.basename().split('-') + return parts[1] + + def repository_name(self): + # Returns the canonical name of the Bazel repository for this package. + canonical = 'pypi__{}_{}'.format(self.distribution(), self.version()) + # Escape any illegal characters with underscore. + return re.sub('[-.]', '_', canonical) + +parser = argparse.ArgumentParser( + description='Import Python dependencies into Bazel.') + +parser.add_argument('--name', action='store', + help=('The namespace of the import.')) + +parser.add_argument('--input', action='store', + help=('The requirements.txt file to import.')) + +parser.add_argument('--output', action='store', + help=('The requirements.bzl file to export.')) + +parser.add_argument('--directory', action='store', + help=('The directory into which to put .whl files.')) + + +def main(): + args = parser.parse_args() + + # https://github.com/pypa/pip/blob/9.0.1/pip/__init__.py#L209 + if pip_main(["wheel", "-w", args.directory, "-r", args.input]): + sys.exit(1) + + # Enumerate the .whl files we downloaded. + def list_whls(): + dir = args.directory + '/' + for root, unused_dirnames, filenames in os.walk(dir): + for fname in filenames: + if fname.endswith('.whl'): + yield os.path.join(root, fname) + + def whl_library(wheel): + # Indentation here matters. whl_library must be within the scope + # of the function below. We also avoid reimporting an existing WHL. + return """ + if "{repo_name}" not in native.existing_rules(): + whl_library( + name = "{repo_name}", + whl = "@{name}//:{path}", + requirements = "@{name}//:requirements.bzl", + )""".format(name=args.name, repo_name=wheel.repository_name(), + path=wheel.basename()) + + whls = [Wheel(path) for path in list_whls()] + + with open(args.output, 'w') as f: + f.write("""\ +# Install pip requirements. +# +# Generated from {input} + +load("@io_bazel_rules_python//python:whl.bzl", "whl_library") + +def pip_install(): + {whl_libraries} + +_packages = {{ + {mappings} +}} + +all_packages = _packages.values() + +def package(name): + name = name.replace("-", "_") + return _packages[name] +""".format(input=args.input, + whl_libraries='\n'.join(map(whl_library, whls)), + mappings=','.join([ + '"%s": "@%s//:pkg"' % (wheel.distribution(), wheel.repository_name()) + for wheel in whls + ]))) + +if __name__ == '__main__': + main() diff --git a/rules_python/whl.py b/rules_python/whl.py new file mode 100644 index 0000000..d47364a --- /dev/null +++ b/rules_python/whl.py @@ -0,0 +1,131 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""The whl modules defines classes for interacting with Python packages.""" + +import argparse +import json +import os +import re +import zipfile + + +class Wheel(object): + + def __init__(self, path): + self._path = path + + def path(self): + return self._path + + def basename(self): + return os.path.basename(self.path()) + + def distribution(self): + # See https://www.python.org/dev/peps/pep-0427/#file-name-convention + parts = self.basename().split('-') + return parts[0] + + def version(self): + # See https://www.python.org/dev/peps/pep-0427/#file-name-convention + parts = self.basename().split('-') + return parts[1] + + def repository_name(self): + # Returns the canonical name of the Bazel repository for this package. + canonical = 'pypi__{}_{}'.format(self.distribution(), self.version()) + # Escape any illegal characters with underscore. + return re.sub('[-.]', '_', canonical) + + def _dist_info(self): + # Return the name of the dist-info directory within the .whl file. + # e.g. google_cloud-0.27.0-py2.py3-none-any.whl -> + # google_cloud-0.27.0.dist-info + return '{}-{}.dist-info'.format(self.distribution(), self.version()) + + def metadata(self): + # Extract the structured data from metadata.json in the WHL's dist-info + # directory. + with zipfile.ZipFile(self.path(), 'r') as whl: + with whl.open(os.path.join(self._dist_info(), 'metadata.json')) as f: + return json.loads(f.read()) + + def name(self): + return self.metadata().get('name') + + def dependencies(self): + # TODO(mattmoor): Is there a schema to follow for this? + run_requires = self.metadata().get('run_requires', []) + for requirement in run_requires: + if 'extra' in requirement: + # TODO(mattmoor): What's the best way to support "extras"? + # https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras + continue + if 'environment' in requirement: + # TODO(mattmoor): What's the best way to support "environment"? + # This typically communicates things like python version (look at + # "wheel" for a good example) + continue + requires = requirement.get('requires', []) + for entry in requires: + # Strip off any trailing versioning data. + parts = entry.split(' ', 1) + yield parts[0] + + def expand(self, directory): + with zipfile.ZipFile(self.path(), 'r') as whl: + whl.extractall(directory) + + +parser = argparse.ArgumentParser( + description='Unpack a WHL file as a py_library.') + +parser.add_argument('--whl', action='store', + help=('The .whl file we are expanding.')) + +parser.add_argument('--requirements', action='store', + help='The pip_import from which to draw dependencies.') + +parser.add_argument('--directory', action='store', default='.', + help='The directory into which to expand things.') + +def main(): + args = parser.parse_args() + whl = Wheel(args.whl) + + # Extract the files into the current directory + whl.expand(args.directory) + + with open(os.path.join(args.directory, 'BUILD'), 'w') as f: + f.write(""" +package(default_visibility = ["//visibility:public"]) + +load("{requirements}", "package") + +py_library( + name = "pkg", + srcs = glob(["**/*.py"]), + data = glob(["**/*"], exclude=["**/*.py", "**/* *", "BUILD", "WORKSPACE"]), + # This makes this directory a top-level in the python import + # search path for anything that depends on this. + imports = ["."], + deps = [{dependencies}], + )""".format( + requirements=args.requirements, + dependencies=','.join([ + 'package("%s")' % d + for d in whl.dependencies() + ]))) + +if __name__ == '__main__': + main() diff --git a/rules_python/whl_test.py b/rules_python/whl_test.py new file mode 100644 index 0000000..1b06821 --- /dev/null +++ b/rules_python/whl_test.py @@ -0,0 +1,57 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import unittest + +from rules_python import whl + + +def TestData(name): + return os.path.join(os.environ['TEST_SRCDIR'], name) + + +class WheelTest(unittest.TestCase): + + def test_grpc_whl(self): + td = TestData('grpc_whl/file/grpcio-1.6.0-cp27-cp27m-manylinux1_i686.whl') + wheel = whl.Wheel(td) + self.assertEqual(wheel.name(), 'grpcio') + self.assertEqual(wheel.distribution(), 'grpcio') + self.assertEqual(wheel.version(), '1.6.0') + self.assertEqual(set(wheel.dependencies()), + set(['enum34', 'futures', 'protobuf', 'six'])) + self.assertEqual('pypi__grpcio_1_6_0', wheel.repository_name()) + + def test_futures_whl(self): + td = TestData('futures_whl/file/futures-3.1.1-py2-none-any.whl') + wheel = whl.Wheel(td) + self.assertEqual(wheel.name(), 'futures') + self.assertEqual(wheel.distribution(), 'futures') + self.assertEqual(wheel.version(), '3.1.1') + self.assertEqual(set(wheel.dependencies()), set()) + self.assertEqual('pypi__futures_3_1_1', wheel.repository_name()) + + def test_mock_whl(self): + td = TestData('mock_whl/file/mock-2.0.0-py2.py3-none-any.whl') + wheel = whl.Wheel(td) + self.assertEqual(wheel.name(), 'mock') + self.assertEqual(wheel.distribution(), 'mock') + self.assertEqual(wheel.version(), '2.0.0') + self.assertEqual(set(wheel.dependencies()), + set(['pbr', 'six'])) + self.assertEqual('pypi__mock_2_0_0', wheel.repository_name()) + +if __name__ == '__main__': + unittest.main() diff --git a/tools/BUILD b/tools/BUILD new file mode 100644 index 0000000..c70f030 --- /dev/null +++ b/tools/BUILD @@ -0,0 +1,19 @@ +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +package(default_visibility = ["//visibility:public"]) + +licenses(["notice"]) # Apache 2.0 + +# This is generated and updated by ./update_piptool.sh +exports_files(["piptool.par"]) diff --git a/tools/piptool.par b/tools/piptool.par Binary files differnew file mode 100755 index 0000000..86fed6d --- /dev/null +++ b/tools/piptool.par diff --git a/update_docs.sh b/update_docs.sh new file mode 100755 index 0000000..bb9dec9 --- /dev/null +++ b/update_docs.sh @@ -0,0 +1,21 @@ +#!/bin/bash + +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euo pipefail + +bazel build //docs/... +unzip -d docs/ -o bazel-bin/docs/docs-md-skydoc.zip +unzip -d docs/ -o bazel-bin/docs/docs-html-skydoc.zip diff --git a/update_piptool.sh b/update_piptool.sh new file mode 100755 index 0000000..7eb8450 --- /dev/null +++ b/update_piptool.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euo pipefail + +bazel build //rules_python:piptool.par +cp bazel-bin/rules_python/piptool.par tools/piptool.par diff --git a/update_tools.sh b/update_tools.sh new file mode 100755 index 0000000..7eb8450 --- /dev/null +++ b/update_tools.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +# Copyright 2017 The Bazel Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euo pipefail + +bazel build //rules_python:piptool.par +cp bazel-bin/rules_python/piptool.par tools/piptool.par |