pw_env_setup: Move config to pigweed.json

Move the environment config to pigweed.json. Since this was already in
JSON, allow it to be at the top of the config file or at
'.["pw"]["pw_env_setup"]'. Add some handling for common mistakes as
well.

Bug: b/268338592
Change-Id: I9031cb9cf67b756465c9928deceb5b32c5eeb230
Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/137210
Reviewed-by: Ted Pudlik <tpudlik@google.com>
Commit-Queue: Rob Mohr <mohrr@google.com>

5 files changed, 111 insertions, 58 deletions

diff --git a/bootstrap.bat b/bootstrap.bat
index a61d2233e..fbe1300ec 100644
--- a/bootstrap.bat
+++ b/bootstrap.bat
@@ -110,7 +110,6 @@ call "%python%" "%PW_ROOT%\pw_env_setup\py\pw_env_setup\env_setup.py" ^
     --pw-root "%PW_ROOT%" ^
     --shell-file "%shell_file%" ^
     --install-dir "%_PW_ACTUAL_ENVIRONMENT_ROOT%" ^
-    --config-file "%PW_ROOT%/pw_env_setup/config.json" ^
     --project-root "%PW_PROJECT_ROOT%"
 goto activate_shell
 
diff --git a/bootstrap.sh b/bootstrap.sh
index a615fc501..3c1c5019b 100644
--- a/bootstrap.sh
+++ b/bootstrap.sh
@@ -81,7 +81,7 @@ SETUP_SH="$_PW_ACTUAL_ENVIRONMENT_ROOT/activate.sh"
 if [ "$(basename "$_PW_BOOTSTRAP_PATH")" = "bootstrap.sh" ] || \
   [ ! -f "$SETUP_SH" ] || \
   [ ! -s "$SETUP_SH" ]; then
-  pw_bootstrap --shell-file "$SETUP_SH" --install-dir "$_PW_ACTUAL_ENVIRONMENT_ROOT" --config-file "$PW_ROOT/pw_env_setup/config.json"
+  pw_bootstrap --shell-file "$SETUP_SH" --install-dir "$_PW_ACTUAL_ENVIRONMENT_ROOT"
   pw_finalize bootstrap "$SETUP_SH"
 else
   pw_activate
diff --git a/pigweed.json b/pigweed.json
index 100160c2d..40bfd2083 100644
--- a/pigweed.json
+++ b/pigweed.json
@@ -47,6 +47,24 @@
           "function": "main"
         }
       }
+    },
+    "pw_env_setup": {
+      "root_variable": "PW_ROOT",
+      "rosetta": "allow",
+      "gni_file": "build_overrides/pigweed_environment.gni",
+      "cipd_package_files": [
+        "pw_env_setup/py/pw_env_setup/cipd_setup/upstream.json"
+      ],
+      "virtualenv": {
+        "gn_targets": ["python.install"],
+        "gn_root": ".",
+        "requirements": [
+          "pw_env_setup/py/pw_env_setup/virtualenv_setup/pigweed_upstream_requirements.txt"
+        ],
+        "constraints": [
+          "pw_env_setup/py/pw_env_setup/virtualenv_setup/constraint.list"
+        ]
+      }
     }
   }
 }
diff --git a/pw_env_setup/docs.rst b/pw_env_setup/docs.rst
index 85cd7cb95..589d8b21b 100644
--- a/pw_env_setup/docs.rst
+++ b/pw_env_setup/docs.rst
@@ -204,20 +204,20 @@ process. To check for this add the following.
 Downstream Projects Using Different Packages
 ********************************************
 Projects depending on Pigweed but using additional or different packages should
-copy the Pigweed `sample project`'s ``bootstrap.sh`` and ``config.json`` and
+copy the Pigweed `sample project`'s ``bootstrap.sh`` and ``pigweed.json`` and
 update the call to ``pw_bootstrap``. Search for "downstream" for other places
 that may require changes, like setting the ``PW_ROOT`` and ``PW_PROJECT_ROOT``
-environment variables. Explanations of parts of ``config.json`` are described
+environment variables. Explanations of parts of ``pigweed.json`` are described
 here.
 
 .. _sample project: https://pigweed.googlesource.com/pigweed/sample_project/+/HEAD
 
-``root_variable``
+``pw.pw_env_setup.root_variable``
   Variable used to point to the root of the source tree. Optional, can always
   use ``PW_PROJECT_ROOT`` instead. (That variable will be set regardless of
   whether this is provided.)
 
-``cipd_package_files``
+``pw.pw_env_setup.cipd_package_files``
   CIPD package file. JSON file consisting of a list of additional CIPD package
   files to import and a list of dictionaries with "path", "platforms", "subdir",
   "tags", and "version_file" keys. Both top-level lists are optional. An
@@ -250,60 +250,60 @@ here.
      ]
    }
 
-``virtualenv.gn_args``
+``pw.pw_env_setup.virtualenv.gn_args``
   Any necessary GN args to be used when installing Python packages.
 
-``virtualenv.gn_targets``
+``pw.pw_env_setup.virtualenv.gn_targets``
   Target for installing Python packages. Downstream projects will need to
   create targets to install their packages or only use Pigweed Python packages.
 
-``virtualenv.gn_root``
+``pw.pw_env_setup.virtualenv.gn_root``
   The root directory of your GN build tree, relative to ``PW_PROJECT_ROOT``.
   This is the directory your project's ``.gn`` file is located in. If you're
   only installing Pigweed Python packages, use the location of the Pigweed
   submodule.
 
-``virtualenv.requirements``
+``pw.pw_env_setup.virtualenv.requirements``
   A list of Python Pip requirements files for installing into the Pigweed
   virtualenv. Each file will be passed as additional ``--requirement`` argument
   to a single ```pip install`` at the beginning of bootstrap's ``Python
   environment`` setup stage. See the `Requirements Files documentation`_ for
   details on what can be specified using requirements files.
 
-``virtualenv.constraints``
+``pw.pw_env_setup.virtualenv.constraints``
   A list of Python Pip constraints files. These constraints will be passed to
   every ``pip`` invocation as an additional ``--constraint`` argument during
   bootstrap.  virtualenv. See the `Constraints Files documentation`_ for details
   on formatting.
 
-``virtualenv.system_packages``
+``pw.pw_env_setup.virtualenv.system_packages``
   A boolean value that can be used the give the Python virtual environment
   access to the system site packages. Defaults to ``false``.
 
-``optional_submodules``
+``pw.pw_env_setup.optional_submodules``
   By default environment setup will check that all submodules are present in
   the checkout. Any submodules in this list are excluded from that check.
 
-``required_submodules``
+``pw.pw_env_setup.required_submodules``
   If this is specified instead of ``optional_submodules`` bootstrap will only
   complain if one of the required submodules is not present. Combining this
   with ``optional_submodules`` is not supported.
 
-``pw_packages``
+``pw.pw_env_setup.pw_packages``
   A list of packages to install using :ref:`pw_package <module-pw_package>`
   after the rest of bootstrap completes.
 
-``gni_file``
+``pw.pw_env_setup.gni_file``
   Location to write a ``.gni`` file containing paths to many things within the
   environment directory. Defaults to
   ``build_overrides/pigweed_environment.gni``.
 
-``json_file``
+``pw.pw_env_setup.json_file``
   Location to write a ``.json`` file containing step-by-step modifications to
   the environment, for reading by tools that don't inherit an environment from
   a sourced ``bootstrap.sh``.
 
-``rosetta``
+``pw.pw_env_setup.rosetta``
   Whether to use Rosetta to use amd64 packages on arm64 Macs. Accepted values
   are  ``never``, ``allow``, and ``force``. For now, ``allow`` means ``force``.
   At some point in the future ``allow`` will be changed to mean ``never``.
@@ -313,33 +313,37 @@ An example of a config file is below.
 .. code-block:: json
 
    {
-     "root_variable": "EXAMPLE_ROOT",
-     "cipd_package_files": [
-       "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json",
-       "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json"
-       "tools/myprojectname.json"
-     ],
-     "virtualenv": {
-       "gn_root": ".",
-       "gn_targets": [
-         ":python.install",
-       ],
-       "system_packages": false
-     },
-     "pw_packages": [],
-     "optional_submodules": [
-       "optional/submodule/one",
-       "optional/submodule/two"
-     ],
-     "gni_file": "tools/environment.gni",
-     "json_file": "tools/environment.json",
-     "rosetta": "allow"
+     "pw": {
+       "pw_env_setup": {
+         "root_variable": "EXAMPLE_ROOT",
+         "cipd_package_files": [
+           "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json",
+           "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json"
+           "tools/myprojectname.json"
+         ],
+         "virtualenv": {
+           "gn_root": ".",
+           "gn_targets": [
+             ":python.install",
+           ],
+           "system_packages": false
+         },
+         "pw_packages": [],
+         "optional_submodules": [
+           "optional/submodule/one",
+           "optional/submodule/two"
+         ],
+         "gni_file": "tools/environment.gni",
+         "json_file": "tools/environment.json",
+         "rosetta": "allow"
+       }
+     }
    }
 
 Only the packages necessary for almost all projects based on Pigweed are
-included in the ``pigweed.json`` file. A number of other files are present in
-that directory for projects that need more than the minimum. Internal-Google
-projects using LUCI should at least include ``luci.json``.
+included in the ``cipd_setup/pigweed.json`` file. A number of other files are
+present in that directory for projects that need more than the minimum.
+Internal-Google projects using LUCI should at least include ``luci.json``.
 
 In case the CIPD packages need to be referenced from other scripts, variables
 like ``PW_${BASENAME}_CIPD_INSTALL_DIR`` point to the CIPD install directories,
@@ -367,8 +371,20 @@ last topologically takes priority. For example, with the file contents below,
 ``d.json``'s entries will appear in ``PATH`` before ``c.json``'s, which will
 appear before ``b.json``'s, which will appear before ``a.json``'s.
 
-``config.json``
-  ``{"cipd_package_files": ["a.json", "b.json", "d.json"], ...}``
+``pigweed.json``
+  .. code-block:: json
+
+     {
+       "pw": {
+         "pw_env_setup": {
+           "cipd_package_files": [
+             "a.json",
+             "b.json",
+             "d.json"
+           ]
+         }
+       }
+     }
 
 ``a.json``
   ``{"package_files": [...]}``
diff --git a/pw_env_setup/py/pw_env_setup/env_setup.py b/pw_env_setup/py/pw_env_setup/env_setup.py
index 7b1966847..8f160a8d7 100755
--- a/pw_env_setup/py/pw_env_setup/env_setup.py
+++ b/pw_env_setup/py/pw_env_setup/env_setup.py
@@ -160,6 +160,11 @@ class MissingSubmodulesError(Exception):
     pass
 
 
+def _assert_sequence(value):
+    assert isinstance(value, (list, tuple))
+    return value
+
+
 # TODO(mohrr) remove disable=useless-object-inheritance once in Python 3.
 # pylint: disable=useless-object-inheritance
 # pylint: disable=too-many-instance-attributes
@@ -224,8 +229,10 @@ class EnvSetup(object):
         self._json_file = json_file
         self._gni_file = None
 
-        self._config_file_name = getattr(config_file, 'name', 'config file')
-        self._env.set('_PW_ENVIRONMENT_CONFIG_FILE', self._config_file_name)
+        self._config_file_name = config_file
+        self._env.set(
+            '_PW_ENVIRONMENT_CONFIG_FILE', os.path.abspath(config_file)
+        )
         if config_file:
             self._parse_config_file(config_file)
 
@@ -269,7 +276,13 @@ class EnvSetup(object):
         return files, warnings
 
     def _parse_config_file(self, config_file):
-        config = json.load(config_file)
+        with open(config_file, 'rb') as ins:
+            config = json.load(ins)
+
+            # While transitioning, allow environment config to be at the top of
+            # the JSON file or at '.["pw"]["pw_env_setup"]'.
+            config = config.get('pw', config)
+            config = config.get('pw_env_setup', config)
 
         self._root_variable = config.pop('root_variable', None)
 
@@ -284,8 +297,12 @@ class EnvSetup(object):
 
         self._gni_file = config.pop('gni_file', None)
 
-        self._optional_submodules.extend(config.pop('optional_submodules', ()))
-        self._required_submodules.extend(config.pop('required_submodules', ()))
+        self._optional_submodules.extend(
+            _assert_sequence(config.pop('optional_submodules', ()))
+        )
+        self._required_submodules.extend(
+            _assert_sequence(config.pop('required_submodules', ()))
+        )
 
         if self._optional_submodules and self._required_submodules:
             raise ValueError(
@@ -296,10 +313,10 @@ class EnvSetup(object):
 
         self._cipd_package_file.extend(
             os.path.join(self._project_root, x)
-            for x in config.pop('cipd_package_files', ())
+            for x in _assert_sequence(config.pop('cipd_package_files', ()))
         )
 
-        for pkg in config.pop('pw_packages', ()):
+        for pkg in _assert_sequence(config.pop('pw_packages', ())):
             self._pw_packages.append(pkg)
 
         virtualenv = config.pop('virtualenv', {})
@@ -309,23 +326,27 @@ class EnvSetup(object):
         else:
             root = self._project_root
 
-        for target in virtualenv.pop('gn_targets', ()):
+        for target in _assert_sequence(virtualenv.pop('gn_targets', ())):
             self._virtualenv_gn_targets.append(
                 virtualenv_setup.GnTarget('{}#{}'.format(root, target))
             )
 
-        self._virtualenv_gn_args = virtualenv.pop('gn_args', ())
+        self._virtualenv_gn_args = _assert_sequence(
+            virtualenv.pop('gn_args', ())
+        )
 
         self._virtualenv_system_packages = virtualenv.pop(
             'system_packages', False
         )
 
-        for req_txt in virtualenv.pop('requirements', ()):
+        for req_txt in _assert_sequence(virtualenv.pop('requirements', ())):
             self._virtualenv_requirements.append(
                 os.path.join(self._project_root, req_txt)
             )
 
-        for constraint_txt in virtualenv.pop('constraints', ()):
+        for constraint_txt in _assert_sequence(
+            virtualenv.pop('constraints', ())
+        ):
             self._virtualenv_constraints.append(
                 os.path.join(self._project_root, constraint_txt)
             )
@@ -791,9 +812,8 @@ def parse(argv=None):
 
     parser.add_argument(
         '--config-file',
-        help='JSON file describing CIPD and virtualenv requirements.',
-        type=argparse.FileType('r'),
-        required=True,
+        help='Path to pigweed.json file.',
+        default=os.path.join(project_root, 'pigweed.json'),
     )
 
     parser.add_argument(