Rebase 'studio-master-dev' on 'master' and squash

studio-master-dev has some customizations on top of AOSP master so we
cannot simply track AOSP master. This change rebase studio-master-dev
on top of the tip of AOSP master while preserving customizations in
studio-master-dev and resolving conflicts along the way.

The change essentially updates all these scripts to be runnable with
python3, which is required with the latest repo tool.

Bug: 149266692
Test: manual
Change-Id: Ic21520839ab47a7b14b961b9799c89ca0f1509f3

29 files changed, 2961 insertions, 685 deletions

diff --git a/README.md b/README.md
index 417f051..f75940d 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,4 @@
-# AOSP Presubmit Hooks
+# AOSP Preupload Hooks
 
 [TOC]
 
@@ -49,12 +49,15 @@ settings on top.
 
 ## PREUPLOAD.cfg
 
-This file are checked in the top of a specific git repository.  Stacking them
+This file is checked in the top of a specific git repository.  Stacking them
 in subdirectories (to try and override parent settings) is not supported.
 
 ## Example
 
 ```
+# Per-project `repo upload` hook settings.
+# https://android.googlesource.com/platform/tools/repohooks
+
 [Options]
 ignore_merged_commits = true
 
@@ -128,13 +131,20 @@ The key can be any name (as long as the syntax is valid), as can the program
 that is executed. The key is used as the name of the hook for reporting purposes,
 so it should be at least somewhat descriptive.
 
+Whitespace in the key name is OK!
+
+The keys must be unique as duplicates will silently clobber earlier values.
+
+You do not need to send stderr to stdout.  The tooling will take care of
+merging them together for you automatically.
+
 ```
 [Hook Scripts]
-my_first_hook = program --gogog ${PREUPLOAD_FILES}
-another_hook = funtimes --i-need "some space" ${PREUPLOAD_FILES}
-some_fish = linter --ate-a-cat ${PREUPLOAD_FILES}
-some_cat = formatter --cat-commit ${PREUPLOAD_COMMIT}
-some_dog = tool --no-cat-in-commit-message ${PREUPLOAD_COMMIT_MESSAGE}
+my first hook = program --gogog ${PREUPLOAD_FILES}
+another hook = funtimes --i-need "some space" ${PREUPLOAD_FILES}
+some fish = linter --ate-a-cat ${PREUPLOAD_FILES}
+some cat = formatter --cat-commit ${PREUPLOAD_COMMIT}
+some dog = tool --no-cat-in-commit-message ${PREUPLOAD_COMMIT_MESSAGE}
 ```
 
 ## [Builtin Hooks]
@@ -149,14 +159,20 @@ canned hooks already included geared towards AOSP style guidelines.
 * `commit_msg_buganizer_field`: Require a valid Bug: or Fixes: field.
 * `commit_msg_bug_field`: Require a valid `Bug:` line.
 * `commit_msg_changeid_field`: Require a valid `Change-Id:` Gerrit line.
+* `commit_msg_prebuilt_apk_fields`: Require badging and build information for
+  prebuilt APKs.
 * `commit_msg_test_field`: Require a `Test:` line.
 * `cpplint`: Run through the cpplint tool (for C++ code).
 * `gofmt`: Run Go code through `gofmt`.
 * `google_java_format`: Run Java code through
   [`google-java-format`](https://github.com/google/google-java-format)
 * `jsonlint`: Verify JSON code is sane.
-* `pylint`: Run Python code through `pylint`.
+* `pylint`: Alias of `pylint2`.  Will change to `pylint3` by end of 2019.
+* `pylint2`: Run Python code through `pylint` using Python 2.
+* `pylint3`: Run Python code through `pylint` using Python 3.
 * `xmllint`: Run XML code through `xmllint`.
+* `android_test_mapping_format`: Validate TEST_MAPPING files in Android source
+  code. Refer to go/test-mapping for more details.
 
 Note: Builtin hooks tend to match specific filenames (e.g. `.json`).  If no
 files match in a specific commit, then the hook will be skipped for that commit.
@@ -195,6 +211,7 @@ executables can be overridden through `[Tool Paths]`.  This is helpful to
 provide consistent behavior for developers across different OS and Linux
 distros/versions.  The following tools are recognized:
 
+* `bpfmt`: used for the `bpfmt` builtin hook.
 * `clang-format`: used for the `clang_format` builtin hook.
 * `cpplint`: used for the `cpplint` builtin hook.
 * `git-clang-format`: used for the `clang_format` builtin hook.
@@ -202,6 +219,8 @@ distros/versions.  The following tools are recognized:
 * `google-java-format`: used for the `google_java_format` builtin hook.
 * `google-java-format-diff`: used for the `google_java_format` builtin hook.
 * `pylint`: used for the `pylint` builtin hook.
+* `android-test-mapping-format`: used for the `android_test_mapping_format`
+  builtin hook.
 
 See [Placeholders](#Placeholders) for variables you can expand automatically.
 
@@ -224,9 +243,15 @@ These are notes for people updating the `pre-upload.py` hook itself:
 * New hooks can be added in `rh/hooks.py`.  Be sure to keep the list up-to-date
   with the documentation in this file.
 
+### Warnings
+
+If the return code of a hook is 77, then it is assumed to be a warning.  The
+output will be printed to the terminal, but uploading will still be allowed
+without a bypass being required.
+
 ## TODO/Limitations
 
-* `pylint` should support per-repo pylintrc files.
+* `pylint` should support per-directory pylintrc files.
 * Some checkers operate on the files as they exist in the filesystem.  This is
   not easy to fix because the linters require not just the modified file but the
   entire repo in order to perform full checks.  e.g. `pylint` needs to know what
diff --git a/pre-upload.py b/pre-upload.py
index eab9045..76d49e5 100755
--- a/pre-upload.py
+++ b/pre-upload.py
@@ -26,22 +26,33 @@ import argparse
 import os
 import sys
 
-try:
-    __file__
-except NameError:
-    # Work around repo until it gets fixed.
-    # https://gerrit-review.googlesource.com/75481
-    __file__ = os.path.join(os.getcwd(), 'pre-upload.py')
+
+# Assert some minimum Python versions as we don't test or support any others.
+# We only support Python 2.7, and require 2.7.5+/3.4+ to include signal fix:
+# https://bugs.python.org/issue14173
+if sys.version_info < (2, 7, 5):
+    print('repohooks: error: Python-2.7.5+ is required', file=sys.stderr)
+    sys.exit(1)
+elif sys.version_info.major == 3 and sys.version_info < (3, 4):
+    # We don't actually test <Python-3.6.  Hope for the best!
+    print('repohooks: error: Python-3.4+ is required', file=sys.stderr)
+    sys.exit(1)
+
+
 _path = os.path.dirname(os.path.realpath(__file__))
 if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# We have to import our local modules after the sys.path tweak.  We can't use
+# relative imports because this is an executable program, not a module.
+# pylint: disable=wrong-import-position
 import rh
 import rh.results
 import rh.config
 import rh.git
 import rh.hooks
+import rh.sixish
 import rh.terminal
 import rh.utils
 
@@ -58,20 +69,27 @@ class Output(object):
     RUNNING = COLOR.color(COLOR.YELLOW, 'RUNNING')
     PASSED = COLOR.color(COLOR.GREEN, 'PASSED')
     FAILED = COLOR.color(COLOR.RED, 'FAILED')
-    WARNING = COLOR.color(COLOR.RED, 'WARNING')
+    WARNING = COLOR.color(COLOR.YELLOW, 'WARNING')
 
-    def __init__(self, project_name, num_hooks):
+    def __init__(self, project_name):
         """Create a new Output object for a specified project.
 
         Args:
           project_name: name of project.
-          num_hooks: number of hooks to be run.
         """
         self.project_name = project_name
-        self.num_hooks = num_hooks
+        self.num_hooks = None
         self.hook_index = 0
         self.success = True
 
+    def set_num_hooks(self, num_hooks):
+        """Keep track of how many hooks we'll be running.
+
+        Args:
+          num_hooks: number of hooks to be run.
+        """
+        self.num_hooks = num_hooks
+
     def commit_start(self, commit, commit_summary):
         """Emit status for new commit.
 
@@ -95,19 +113,39 @@ class Output(object):
         rh.terminal.print_status_line(status_line)
 
     def hook_error(self, hook_name, error):
-        """Print an error.
+        """Print an error for a single hook.
 
         Args:
           hook_name: name of the hook.
           error: error string.
         """
-        status_line = '[%s] %s' % (self.FAILED, hook_name)
+        self.error(hook_name, error)
+
+    def hook_warning(self, hook_name, warning):
+        """Print a warning for a single hook.
+
+        Args:
+          hook_name: name of the hook.
+          warning: warning string.
+        """
+        status_line = '[%s] %s' % (self.WARNING, hook_name)
+        rh.terminal.print_status_line(status_line, print_newline=True)
+        print(warning, file=sys.stderr)
+
+    def error(self, header, error):
+        """Print a general error.
+
+        Args:
+          header: A unique identifier for the source of this error.
+          error: error string.
+        """
+        status_line = '[%s] %s' % (self.FAILED, header)
         rh.terminal.print_status_line(status_line, print_newline=True)
         print(error, file=sys.stderr)
         self.success = False
 
     def finish(self):
-        """Print repohook summary."""
+        """Print summary for all the hooks."""
         status_line = '[%s] repohooks for %s %s' % (
             self.PASSED if self.success else self.FAILED,
             self.project_name,
@@ -123,19 +161,26 @@ def _process_hook_results(results):
 
     Returns:
       error output if an error occurred, otherwise None
+      warning output if an error occurred, otherwise None
     """
     if not results:
-        return None
+        return (None, None)
 
-    ret = ''
+    error_ret = ''
+    warning_ret = ''
     for result in results:
         if result:
+            ret = ''
             if result.files:
                 ret += '  FILES: %s' % (result.files,)
             lines = result.error.splitlines()
             ret += '\n'.join('    %s' % (x,) for x in lines)
+            if result.is_warning():
+                warning_ret += ret
+            else:
+                error_ret += ret
 
-    return ret or None
+    return (error_ret or None, warning_ret or None)
 
 
 def _get_project_config():
@@ -153,13 +198,7 @@ def _get_project_config():
         # Load the config for this git repo.
         '.',
     )
-    try:
-        config = rh.config.PreSubmitConfig(paths=paths,
-                                           global_paths=global_paths)
-    except rh.config.ValidationError as e:
-        print('invalid config file: %s' % (e,), file=sys.stderr)
-        sys.exit(1)
-    return config
+    return rh.config.PreUploadConfig(paths=paths, global_paths=global_paths)
 
 
 def _attempt_fixes(fixup_func_list, commit_list):
@@ -193,14 +232,13 @@ def _attempt_fixes(fixup_func_list, commit_list):
               'attempting to upload again.\n', file=sys.stderr)
 
 
-def _run_project_hooks(project_name, proj_dir=None,
-                       commit_list=None):
-    """For each project run its project specific hook from the hooks dictionary.
+def _run_project_hooks_in_cwd(project_name, proj_dir, output, commit_list=None):
+    """Run the project-specific hooks in the cwd.
 
     Args:
-      project_name: The name of project to run hooks for.
-      proj_dir: If non-None, this is the directory the project is in.  If None,
-          we'll ask repo.
+      project_name: The name of this project.
+      proj_dir: The directory for this project (for passing on in metadata).
+      output: Helper for summarizing output/errors to the user.
       commit_list: A list of commits to run hooks against.  If None or empty
           list then we'll automatically get the list of commits that would be
           uploaded.
@@ -208,40 +246,29 @@ def _run_project_hooks(project_name, proj_dir=None,
     Returns:
       False if any errors were found, else True.
     """
-    if proj_dir is None:
-        cmd = ['repo', 'forall', project_name, '-c', 'pwd']
-        result = rh.utils.run_command(cmd, capture_output=True)
-        proj_dirs = result.output.split()
-        if len(proj_dirs) == 0:
-            print('%s cannot be found.' % project_name, file=sys.stderr)
-            print('Please specify a valid project.', file=sys.stderr)
-            return 0
-        if len(proj_dirs) > 1:
-            print('%s is associated with multiple directories.' % project_name,
-                  file=sys.stderr)
-            print('Please specify a directory to help disambiguate.',
-                  file=sys.stderr)
-            return 0
-        proj_dir = proj_dirs[0]
-
-    pwd = os.getcwd()
-    # Hooks assume they are run from the root of the project.
-    os.chdir(proj_dir)
+    try:
+        config = _get_project_config()
+    except rh.config.ValidationError as e:
+        output.error('Loading config files', str(e))
+        return False
 
     # If the repo has no pre-upload hooks enabled, then just return.
-    config = _get_project_config()
     hooks = list(config.callable_hooks())
     if not hooks:
         return True
 
+    output.set_num_hooks(len(hooks))
+
     # Set up the environment like repo would with the forall command.
     try:
         remote = rh.git.get_upstream_remote()
         upstream_branch = rh.git.get_upstream_branch()
     except rh.utils.RunCommandError as e:
-        print('upstream remote cannot be found: %s' % (e,), file=sys.stderr)
-        print('Did you run repo start?', file=sys.stderr)
-        sys.exit(1)
+        output.error('Upstream remote/tracking branch lookup',
+                     '%s\nDid you run repo start?  Is your HEAD detached?' %
+                     (e,))
+        return False
+
     os.environ.update({
         'REPO_LREV': str(rh.git.get_commit_for_ref(upstream_branch)),
         'REPO_PATH': str(proj_dir),
@@ -250,7 +277,6 @@ def _run_project_hooks(project_name, proj_dir=None,
         'REPO_RREV': str(rh.git.get_remote_revision(upstream_branch, remote)),
     })
 
-    output = Output(project_name, len(hooks))
     project = rh.Project(name=project_name, dir=proj_dir, remote=remote)
 
     if not commit_list:
@@ -292,7 +318,7 @@ def _run_project_hooks(project_name, proj_dir=None,
         os.environ['PREUPLOAD_COMMIT'] = commit
         diff = rh.git.get_affected_files(commit)
         desc = rh.git.get_commit_desc(commit)
-        os.environ['PREUPLOAD_COMMIT_MESSAGE'] = desc
+        rh.sixish.setenv('PREUPLOAD_COMMIT_MESSAGE', desc)
 
         commit_summary = desc.split('\n', 1)[0]
         output.commit_start(commit=commit, commit_summary=commit_summary)
@@ -300,10 +326,13 @@ def _run_project_hooks(project_name, proj_dir=None,
         for name, hook in hooks:
             output.hook_start(name)
             hook_results = hook(project, commit, desc, diff)
-            error = _process_hook_results(hook_results)
-            if error:
-                ret = False
-                output.hook_error(name, error)
+            (error, warning) = _process_hook_results(hook_results)
+            if error or warning:
+                if warning:
+                    output.hook_warning(name, warning)
+                if error:
+                    ret = False
+                    output.hook_error(name, error)
                 for result in hook_results:
                     if result.fixup_func:
                         fixup_func_list.append((name, commit,
@@ -312,11 +341,52 @@ def _run_project_hooks(project_name, proj_dir=None,
     if fixup_func_list:
         _attempt_fixes(fixup_func_list, commit_list)
 
-    output.finish()
-    os.chdir(pwd)
     return ret
 
 
+def _run_project_hooks(project_name, proj_dir=None, commit_list=None):
+    """Run the project-specific hooks in |proj_dir|.
+
+    Args:
+      project_name: The name of project to run hooks for.
+      proj_dir: If non-None, this is the directory the project is in.  If None,
+          we'll ask repo.
+      commit_list: A list of commits to run hooks against.  If None or empty
+          list then we'll automatically get the list of commits that would be
+          uploaded.
+
+    Returns:
+      False if any errors were found, else True.
+    """
+    output = Output(project_name)
+
+    if proj_dir is None:
+        cmd = ['repo', 'forall', project_name, '-c', 'pwd']
+        result = rh.utils.run_command(cmd, capture_output=True)
+        proj_dirs = result.output.split()
+        if not proj_dirs:
+            print('%s cannot be found.' % project_name, file=sys.stderr)
+            print('Please specify a valid project.', file=sys.stderr)
+            return False
+        if len(proj_dirs) > 1:
+            print('%s is associated with multiple directories.' % project_name,
+                  file=sys.stderr)
+            print('Please specify a directory to help disambiguate.',
+                  file=sys.stderr)
+            return False
+        proj_dir = proj_dirs[0]
+
+    pwd = os.getcwd()
+    try:
+        # Hooks assume they are run from the root of the project.
+        os.chdir(proj_dir)
+        return _run_project_hooks_in_cwd(project_name, proj_dir, output,
+                                         commit_list=commit_list)
+    finally:
+        output.finish()
+        os.chdir(pwd)
+
+
 def main(project_list, worktree_list=None, **_kwargs):
     """Main function invoked directly by repo.
 
@@ -339,6 +409,10 @@ def main(project_list, worktree_list=None, **_kwargs):
     for project, worktree in zip(project_list, worktree_list):
         if not _run_project_hooks(project, proj_dir=worktree):
             found_error = True
+            # If a repo had failures, add a blank line to help break up the
+            # output.  If there were no failures, then the output should be
+            # very minimal, so we don't add it then.
+            print('', file=sys.stderr)
 
     if found_error:
         color = rh.terminal.Color()
@@ -397,8 +471,8 @@ def direct_main(argv):
         opts.dir = os.path.dirname(os.path.abspath(git_dir))
     elif not os.path.isdir(opts.dir):
         parser.error('Invalid dir: %s' % opts.dir)
-    elif not os.path.isdir(os.path.join(opts.dir, '.git')):
-        parser.error('Not a git directory: %s' % opts.dir)
+    elif not rh.git.is_git_repository(opts.dir):
+        parser.error('Not a git repository: %s' % opts.dir)
 
     # Identify the project if it wasn't specified; this _requires_ the repo
     # tool to be installed and for the project to be part of a repo checkout.
@@ -410,8 +484,7 @@ def direct_main(argv):
     if _run_project_hooks(opts.project, proj_dir=opts.dir,
                           commit_list=opts.commits):
         return 0
-    else:
-        return 1
+    return 1
 
 def has_hook(hooks, name):
     for (n, h) in hooks:
diff --git a/rh/config.py b/rh/config.py
index 6a149d7..6e96fc7 100644
--- a/rh/config.py
+++ b/rh/config.py
@@ -17,7 +17,6 @@
 
 from __future__ import print_function
 
-import ConfigParser
 import functools
 import os
 import shlex
@@ -28,8 +27,10 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# pylint: disable=wrong-import-position
 import rh.hooks
 import rh.shell
+from rh.sixish import configparser
 
 
 class Error(Exception):
@@ -40,7 +41,7 @@ class ValidationError(Error):
     """Config file has unknown sections/keys or other values."""
 
 
-class RawConfigParser(ConfigParser.RawConfigParser):
+class RawConfigParser(configparser.RawConfigParser):
     """Like RawConfigParser but with some default helpers."""
 
     @staticmethod
@@ -51,6 +52,9 @@ class RawConfigParser(ConfigParser.RawConfigParser):
                             (name, cnt_min, cnt_max, cnt,))
         return cnt
 
+    # pylint can't seem to grok our use of *args here.
+    # pylint: disable=arguments-differ
+
     def options(self, section, *args):
         """Return the options in |section| (with default |args|).
 
@@ -60,8 +64,8 @@ class RawConfigParser(ConfigParser.RawConfigParser):
         """
         cnt = self._check_args('options', 2, 3, args)
         try:
-            return ConfigParser.RawConfigParser.options(self, section)
-        except ConfigParser.NoSectionError:
+            return configparser.RawConfigParser.options(self, section)
+        except configparser.NoSectionError:
             if cnt == 1:
                 return args[0]
             raise
@@ -70,8 +74,8 @@ class RawConfigParser(ConfigParser.RawConfigParser):
         """Return the value for |option| in |section| (with default |args|)."""
         cnt = self._check_args('get', 3, 4, args)
         try:
-            return ConfigParser.RawConfigParser.get(self, section, option)
-        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+            return configparser.RawConfigParser.get(self, section, option)
+        except (configparser.NoSectionError, configparser.NoOptionError):
             if cnt == 1:
                 return args[0]
             raise
@@ -80,14 +84,14 @@ class RawConfigParser(ConfigParser.RawConfigParser):
         """Return a list of (key, value) tuples for the options in |section|."""
         cnt = self._check_args('items', 2, 3, args)
         try:
-            return ConfigParser.RawConfigParser.items(self, section)
-        except ConfigParser.NoSectionError:
+            return configparser.RawConfigParser.items(self, section)
+        except configparser.NoSectionError:
             if cnt == 1:
                 return args[0]
             raise
 
 
-class PreSubmitConfig(object):
+class PreUploadConfig(object):
     """Config file used for per-project `repo upload` hooks."""
 
     FILENAME = 'PREUPLOAD.cfg'
@@ -120,7 +124,7 @@ class PreSubmitConfig(object):
                     self.paths.append(path)
                     try:
                         config.read(path)
-                    except ConfigParser.ParsingError as e:
+                    except configparser.ParsingError as e:
                         raise ValidationError('%s: %s' % (path, e))
 
         self.paths = []
diff --git a/rh/config_test.py b/rh/config_test.py
new file mode 100755
index 0000000..794e50f
--- /dev/null
+++ b/rh/config_test.py
@@ -0,0 +1,113 @@
+#!/usr/bin/env python3
+# -*- coding:utf-8 -*-
+# Copyright 2019 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Integration tests for the config module (via PREUPLOAD.cfg)."""
+
+from __future__ import print_function
+
+import argparse
+import os
+import re
+import sys
+
+
+REPOTOOLS = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+REPO_ROOT = os.path.dirname(os.path.dirname(REPOTOOLS))
+
+
+def assertEqual(msg, exp, actual):
+    """Assert |exp| equals |actual|."""
+    assert exp == actual, '%s: expected "%s" but got "%s"' % (msg, exp, actual)
+
+
+def assertEnv(var, value):
+    """Assert |var| is set in the environment as |value|."""
+    assert var in os.environ, '$%s missing in environment' % (var,)
+    assertEqual('env[%s]' % (var,), value, os.environ[var])
+
+
+def check_commit_id(commit):
+    """Check |commit| looks like a git commit id."""
+    assert len(commit) == 40, 'commit "%s" must be 40 chars' % (commit,)
+    assert re.match(r'^[a-f0-9]+$', commit), \
+        'commit "%s" must be all hex' % (commit,)
+
+
+def check_commit_msg(msg):
+    """Check the ${PREUPLOAD_COMMIT_MESSAGE} setting."""
+    assert len(msg) > 1, 'commit message must be at least 2 bytes: %s'
+
+
+def check_repo_root(root):
+    """Check the ${REPO_ROOT} setting."""
+    assertEqual('REPO_ROOT', REPO_ROOT, root)
+
+
+def check_files(files):
+    """Check the ${PREUPLOAD_FILES} setting."""
+    assert files
+
+
+def check_env():
+    """Verify all exported env vars look sane."""
+    assertEnv('REPO_PROJECT', 'platform/tools/repohooks')
+    assertEnv('REPO_PATH', 'tools/repohooks')
+    assertEnv('REPO_REMOTE', 'aosp')
+    check_commit_id(os.environ['REPO_LREV'])
+    print(os.environ['REPO_RREV'])
+    check_commit_id(os.environ['PREUPLOAD_COMMIT'])
+
+
+def get_parser():
+    """Return a command line parser."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--check-env', action='store_true',
+                        help='Check all exported env vars.')
+    parser.add_argument('--commit-id',
+                        help='${PREUPLOAD_COMMIT} setting.')
+    parser.add_argument('--commit-msg',
+                        help='${PREUPLOAD_COMMIT_MESSAGE} setting.')
+    parser.add_argument('--repo-root',
+                        help='${REPO_ROOT} setting.')
+    parser.add_argument('files', nargs='+',
+                        help='${PREUPLOAD_FILES} paths.')
+    return parser
+
+
+def main(argv):
+    """The main entry."""
+    parser = get_parser()
+    opts = parser.parse_args(argv)
+
+    try:
+        if opts.check_env:
+            check_env()
+        if opts.commit_id is not None:
+            check_commit_id(opts.commit_id)
+        if opts.commit_msg is not None:
+            check_commit_msg(opts.commit_msg)
+        if opts.repo_root is not None:
+            check_repo_root(opts.repo_root)
+        check_files(opts.files)
+    except AssertionError as e:
+        print('error: %s' % (e,), file=sys.stderr)
+        return 1
+
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[1:]))
diff --git a/rh/config_unittest.py b/rh/config_unittest.py
index 9547b74..d51afdc 100755
--- a/rh/config_unittest.py
+++ b/rh/config_unittest.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 # -*- coding:utf-8 -*-
 # Copyright 2016 The Android Open Source Project
 #
@@ -29,12 +29,15 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# We have to import our local modules after the sys.path tweak.  We can't use
+# relative imports because this is an executable program, not a module.
+# pylint: disable=wrong-import-position
 import rh.hooks
 import rh.config
 
 
-class PreSubmitConfigTests(unittest.TestCase):
-    """Tests for the PreSubmitConfig class."""
+class PreUploadConfigTests(unittest.TestCase):
+    """Tests for the PreUploadConfig class."""
 
     def setUp(self):
         self.tempdir = tempfile.mkdtemp()
@@ -45,7 +48,7 @@ class PreSubmitConfigTests(unittest.TestCase):
     def _write_config(self, data, filename=None):
         """Helper to write out a config file for testing."""
         if filename is None:
-            filename = rh.config.PreSubmitConfig.FILENAME
+            filename = rh.config.PreUploadConfig.FILENAME
         path = os.path.join(self.tempdir, filename)
         with open(path, 'w') as fp:
             fp.write(data)
@@ -53,16 +56,16 @@ class PreSubmitConfigTests(unittest.TestCase):
     def _write_global_config(self, data):
         """Helper to write out a global config file for testing."""
         self._write_config(
-            data, filename=rh.config.PreSubmitConfig.GLOBAL_FILENAME)
+            data, filename=rh.config.PreUploadConfig.GLOBAL_FILENAME)
 
     def testMissing(self):
         """Instantiating a non-existent config file should be fine."""
-        rh.config.PreSubmitConfig()
+        rh.config.PreUploadConfig()
 
     def testEmpty(self):
         """Instantiating an empty config file should be fine."""
         self._write_config('')
-        rh.config.PreSubmitConfig(paths=(self.tempdir,))
+        rh.config.PreUploadConfig(paths=(self.tempdir,))
 
     def testValid(self):
         """Verify a fully valid file works."""
@@ -79,30 +82,30 @@ cpplint = --some 'more args'
 [Options]
 ignore_merged_commits = true
 """)
-        rh.config.PreSubmitConfig(paths=(self.tempdir,))
+        rh.config.PreUploadConfig(paths=(self.tempdir,))
 
     def testUnknownSection(self):
         """Reject unknown sections."""
         self._write_config('[BOOGA]')
-        self.assertRaises(rh.config.ValidationError, rh.config.PreSubmitConfig,
+        self.assertRaises(rh.config.ValidationError, rh.config.PreUploadConfig,
                           paths=(self.tempdir,))
 
     def testUnknownBuiltin(self):
         """Reject unknown builtin hooks."""
         self._write_config('[Builtin Hooks]\nbooga = borg!')
-        self.assertRaises(rh.config.ValidationError, rh.config.PreSubmitConfig,
+        self.assertRaises(rh.config.ValidationError, rh.config.PreUploadConfig,
                           paths=(self.tempdir,))
 
     def testEmptyCustomHook(self):
         """Reject empty custom hooks."""
         self._write_config('[Hook Scripts]\nbooga = \t \n')
-        self.assertRaises(rh.config.ValidationError, rh.config.PreSubmitConfig,
+        self.assertRaises(rh.config.ValidationError, rh.config.PreUploadConfig,
                           paths=(self.tempdir,))
 
     def testInvalidIni(self):
         """Reject invalid ini files."""
         self._write_config('[Hook Scripts]\n =')
-        self.assertRaises(rh.config.ValidationError, rh.config.PreSubmitConfig,
+        self.assertRaises(rh.config.ValidationError, rh.config.PreUploadConfig,
                           paths=(self.tempdir,))
 
     def testInvalidString(self):
@@ -110,7 +113,7 @@ ignore_merged_commits = true
         self._write_config("""[Hook Scripts]
 name = script --'bad-quotes
 """)
-        self.assertRaises(rh.config.ValidationError, rh.config.PreSubmitConfig,
+        self.assertRaises(rh.config.ValidationError, rh.config.PreUploadConfig,
                           paths=(self.tempdir,))
 
     def testGlobalConfigs(self):
@@ -122,7 +125,7 @@ commit_msg_test_field = false""")
         self._write_config("""[Builtin Hooks]
 commit_msg_bug_field = false
 commit_msg_test_field = true""")
-        config = rh.config.PreSubmitConfig(paths=(self.tempdir,),
+        config = rh.config.PreUploadConfig(paths=(self.tempdir,),
                                            global_paths=(self.tempdir,))
         self.assertEqual(config.builtin_hooks,
                          ['commit_msg_changeid_field', 'commit_msg_test_field'])
diff --git a/rh/git.py b/rh/git.py
index 8cf2557..cbe8c42 100644
--- a/rh/git.py
+++ b/rh/git.py
@@ -26,6 +26,7 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# pylint: disable=wrong-import-position
 import rh.utils
 
 
@@ -91,18 +92,6 @@ def get_patch(commit):
     return rh.utils.run_command(cmd, capture_output=True).output
 
 
-def _try_utf8_decode(data):
-    """Attempts to decode a string as UTF-8.
-
-    Returns:
-      The decoded Unicode object, or the original string if parsing fails.
-    """
-    try:
-        return unicode(data, 'utf-8', 'strict')
-    except UnicodeDecodeError:
-        return data
-
-
 def get_file_content(commit, path):
     """Returns the content of a file at a specific commit.
 
@@ -117,11 +106,22 @@ def get_file_content(commit, path):
     return rh.utils.run_command(cmd, capture_output=True).output
 
 
-# RawDiffEntry represents a line of raw formatted git diff output.
-RawDiffEntry = rh.utils.collection(
-    'RawDiffEntry',
-    src_mode=0, dst_mode=0, src_sha=None, dst_sha=None,
-    status=None, score=None, src_file=None, dst_file=None, file=None)
+class RawDiffEntry(object):
+    """Representation of a line from raw formatted git diff output."""
+
+    # pylint: disable=redefined-builtin
+    def __init__(self, src_mode=0, dst_mode=0, src_sha=None, dst_sha=None,
+                 status=None, score=None, src_file=None, dst_file=None,
+                 file=None):
+        self.src_mode = src_mode
+        self.dst_mode = dst_mode
+        self.src_sha = src_sha
+        self.dst_sha = dst_sha
+        self.status = status
+        self.score = score
+        self.src_file = src_file
+        self.dst_file = dst_file
+        self.file = file
 
 
 # This regular expression pulls apart a line of raw formatted git diff output.
@@ -144,18 +144,19 @@ def raw_diff(path, target):
     """
     entries = []
 
-    cmd = ['git', 'diff', '-M', '--raw', target]
+    cmd = ['git', 'diff', '--no-ext-diff', '-M', '--raw', target]
     diff = rh.utils.run_command(cmd, cwd=path, capture_output=True).output
     diff_lines = diff.strip().splitlines()
     for line in diff_lines:
         match = DIFF_RE.match(line)
         if not match:
             raise ValueError('Failed to parse diff output: %s' % line)
-        diff = RawDiffEntry(**match.groupdict())
-        diff.src_mode = int(diff.src_mode)
-        diff.dst_mode = int(diff.dst_mode)
-        diff.file = diff.dst_file if diff.dst_file else diff.src_file
-        entries.append(diff)
+        rawdiff = RawDiffEntry(**match.groupdict())
+        rawdiff.src_mode = int(rawdiff.src_mode)
+        rawdiff.dst_mode = int(rawdiff.dst_mode)
+        rawdiff.file = (rawdiff.dst_file
+                        if rawdiff.dst_file else rawdiff.src_file)
+        entries.append(rawdiff)
 
     return entries
 
@@ -196,3 +197,10 @@ def find_repo_root(path=None):
             raise ValueError('Could not locate .repo in %s' % orig_path)
 
     return path
+
+
+def is_git_repository(path):
+    """Returns True if the path is a valid git repository."""
+    cmd = ['git', 'rev-parse', '--resolve-git-dir', os.path.join(path, '.git')]
+    result = rh.utils.run_command(cmd, quiet=True, error_code_ok=True)
+    return result.returncode == 0
diff --git a/rh/hooks.py b/rh/hooks.py
index cff5728..40b34b2 100644
--- a/rh/hooks.py
+++ b/rh/hooks.py
@@ -28,8 +28,10 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
-import rh.results
+# pylint: disable=wrong-import-position
 import rh.git
+import rh.results
+from rh.sixish import string_types
 import rh.utils
 
 
@@ -71,7 +73,7 @@ class Placeholders(object):
             for key, val in replacements.items():
                 var = '${%s}' % (key,)
                 if arg == var:
-                    if isinstance(val, str):
+                    if isinstance(val, string_types):
                         ret.append(val)
                     else:
                         ret.extend(val)
@@ -81,10 +83,9 @@ class Placeholders(object):
                 # If no exact matches, do an inline replacement.
                 def replace(m):
                     val = self.get(m.group(1))
-                    if isinstance(val, str):
+                    if isinstance(val, string_types):
                         return val
-                    else:
-                        return ' '.join(val)
+                    return ' '.join(val)
                 ret.append(re.sub(r'\$\{(%s)\}' % ('|'.join(all_vars),),
                                   replace, arg))
 
@@ -305,6 +306,24 @@ def check_custom(project, commit, _desc, diff, options=None, **kwargs):
                       **kwargs)
 
 
+def check_bpfmt(project, commit, _desc, diff, options=None):
+    """Checks that Blueprint files are formatted with bpfmt."""
+    filtered = _filter_diff(diff, [r'\.bp$'])
+    if not filtered:
+        return None
+
+    bpfmt = options.tool_path('bpfmt')
+    cmd = [bpfmt, '-l'] + options.args((), filtered)
+    ret = []
+    for d in filtered:
+        data = rh.git.get_file_content(commit, d.file)
+        result = _run_command(cmd, input=data)
+        if result.output:
+            ret.append(rh.results.HookResult(
+                'bpfmt', project, commit, error=result.output,
+                files=(d.file,)))
+    return ret
+
 def check_buildifier(project, commit, _desc, diff, options=None):
     """Checks that BUILD files are formatted with buildifier."""
     filtered = _filter_diff(diff, [r'BUILD$', r'BUILD\.bazel$', r'\.bzl$'])
@@ -415,10 +434,10 @@ def check_commit_msg_bug_field(project, commit, desc, _diff, options=None):
             found.append(line)
 
     if not found:
-        error = ('Commit message is missing a "%s:" line.  It must match:\n'
-                 '%s') % (field, regex)
+        error = ('Commit message is missing a "%s:" line.  It must match the\n'
+                 'following case-sensitive regex:\n\n    %s') % (field, regex)
     else:
-        return
+        return None
 
     return [rh.results.HookResult('commit msg: "%s:" check' % (field,),
                                   project, commit, error=error)]
@@ -438,21 +457,80 @@ def check_commit_msg_changeid_field(project, commit, desc, _diff, options=None):
         if check_re.match(line):
             found.append(line)
 
-    if len(found) == 0:
-        error = ('Commit message is missing a "%s:" line.  It must match:\n'
-                 '%s') % (field, regex)
+    if not found:
+        error = ('Commit message is missing a "%s:" line.  It must match the\n'
+                 'following case-sensitive regex:\n\n    %s') % (field, regex)
     elif len(found) > 1:
         error = ('Commit message has too many "%s:" lines.  There can be only '
                  'one.') % (field,)
     else:
-        return
+        return None
 
     return [rh.results.HookResult('commit msg: "%s:" check' % (field,),
                                   project, commit, error=error)]
 
 
-TEST_MSG = """Commit message is missing a "Test:" line.  It must match:
-%s
+PREBUILT_APK_MSG = """Commit message is missing required prebuilt APK
+information.  To generate the information, use the aapt tool to dump badging
+information of the APKs being uploaded, specify where the APK was built, and
+specify whether the APKs are suitable for release:
+
+    for apk in $(find . -name '*.apk' | sort); do
+        echo "${apk}"
+        ${AAPT} dump badging "${apk}" |
+            grep -iE "(package: |sdkVersion:|targetSdkVersion:)" |
+            sed -e "s/' /'\\n/g"
+        echo
+    done
+
+It must match the following case-sensitive multiline regex searches:
+
+    %s
+
+For more information, see go/platform-prebuilt and go/android-prebuilt.
+
+"""
+
+
+def check_commit_msg_prebuilt_apk_fields(project, commit, desc, diff,
+                                         options=None):
+    """Check that prebuilt APK commits contain the required lines."""
+
+    if options.args():
+        raise ValueError('prebuilt apk check takes no options')
+
+    filtered = _filter_diff(diff, [r'\.apk$'])
+    if not filtered:
+        return None
+
+    regexes = [
+        r'^package: .*$',
+        r'^sdkVersion:.*$',
+        r'^targetSdkVersion:.*$',
+        r'^Built here:.*$',
+        (r'^This build IS( NOT)? suitable for'
+         r'( preview|( preview or)? public) release'
+         r'( but IS NOT suitable for public release)?\.$')
+    ]
+
+    missing = []
+    for regex in regexes:
+        if not re.search(regex, desc, re.MULTILINE):
+            missing.append(regex)
+
+    if missing:
+        error = PREBUILT_APK_MSG % '\n    '.join(missing)
+    else:
+        return None
+
+    return [rh.results.HookResult('commit msg: "prebuilt apk:" check',
+                                  project, commit, error=error)]
+
+
+TEST_MSG = """Commit message is missing a "Test:" line.  It must match the
+following case-sensitive regex:
+
+    %s
 
 Your "Test:" description should list how your change is being tested with
 automated tests. If you're submitting a bugfix and there's an obvious unit test
@@ -498,7 +576,7 @@ def check_commit_msg_test_field(project, commit, desc, _diff, options=None):
     if not found:
         error = TEST_MSG % (regex)
     else:
-        return
+        return None
 
     return [rh.results.HookResult('commit msg: "%s:" check' % (field,),
                                   project, commit, error=error)]
@@ -510,7 +588,7 @@ def check_cpplint(project, commit, _desc, diff, options=None):
     # but cpplint would just ignore them.
     filtered = _filter_diff(diff, [r'\.(cc|h|cpp|cu|cuh)$'])
     if not filtered:
-        return
+        return None
 
     cpplint = options.tool_path('cpplint')
     cmd = [cpplint] + options.args(('${PREUPLOAD_FILES}',), filtered)
@@ -521,7 +599,7 @@ def check_gofmt(project, commit, _desc, diff, options=None):
     """Checks that Go files are formatted with gofmt."""
     filtered = _filter_diff(diff, [r'\.go$'])
     if not filtered:
-        return
+        return None
 
     gofmt = options.tool_path('gofmt')
     cmd = [gofmt, '-l'] + options.args((), filtered)
@@ -543,7 +621,7 @@ def check_json(project, commit, _desc, diff, options=None):
 
     filtered = _filter_diff(diff, [r'\.json$'])
     if not filtered:
-        return
+        return None
 
     ret = []
     for d in filtered:
@@ -557,20 +635,35 @@ def check_json(project, commit, _desc, diff, options=None):
     return ret
 
 
-def check_pylint(project, commit, _desc, diff, options=None):
+def _check_pylint(project, commit, _desc, diff, extra_args=None, options=None):
     """Run pylint."""
     filtered = _filter_diff(diff, [r'\.py$'])
     if not filtered:
-        return
+        return None
+
+    if extra_args is None:
+        extra_args = []
 
     pylint = options.tool_path('pylint')
     cmd = [
         get_helper_path('pylint.py'),
         '--executable-path', pylint,
-    ] + options.args(('${PREUPLOAD_FILES}',), filtered)
+    ] + extra_args + options.args(('${PREUPLOAD_FILES}',), filtered)
     return _check_cmd('pylint', project, commit, cmd)
 
 
+def check_pylint2(project, commit, desc, diff, options=None):
+    """Run pylint through Python 2."""
+    return _check_pylint(project, commit, desc, diff, options=options)
+
+
+def check_pylint3(project, commit, desc, diff, options=None):
+    """Run pylint through Python 3."""
+    return _check_pylint(project, commit, desc, diff,
+                         extra_args=['--executable-path=pylint3'],
+                         options=options)
+
+
 def check_xmllint(project, commit, _desc, diff, options=None):
     """Run xmllint."""
     # XXX: Should we drop most of these and probe for <?xml> tags?
@@ -606,7 +699,7 @@ def check_xmllint(project, commit, _desc, diff, options=None):
 
     filtered = _filter_diff(diff, [r'\.(%s)$' % '|'.join(extensions)])
     if not filtered:
-        return
+        return None
 
     # TODO: Figure out how to integrate schema validation.
     # XXX: Should we use python's XML libs instead?
@@ -615,6 +708,21 @@ def check_xmllint(project, commit, _desc, diff, options=None):
     return _check_cmd('xmllint', project, commit, cmd)
 
 
+def check_android_test_mapping(project, commit, _desc, diff, options=None):
+    """Verify Android TEST_MAPPING files are valid."""
+    if options.args():
+        raise ValueError('Android TEST_MAPPING check takes no options')
+    filtered = _filter_diff(diff, [r'(^|.*/)TEST_MAPPING$'])
+    if not filtered:
+        return None
+
+    testmapping_format = options.tool_path('android-test-mapping-format')
+    testmapping_args = ['--commit', commit]
+    cmd = [testmapping_format] + options.args(
+        (project.dir, '${PREUPLOAD_FILES}'), filtered) + testmapping_args
+    return _check_cmd('android-test-mapping-format', project, commit, cmd)
+
+
 # Hooks that projects can opt into.
 # Note: Make sure to keep the top level README.md up to date when adding more!
 BUILTIN_HOOKS = {
@@ -624,12 +732,15 @@ BUILTIN_HOOKS = {
     'commit_msg_buganizer_field': check_commit_msg_buganizer_field,
     'commit_msg_bug_field': check_commit_msg_bug_field,
     'commit_msg_changeid_field': check_commit_msg_changeid_field,
+    'commit_msg_prebuilt_apk_fields': check_commit_msg_prebuilt_apk_fields,
     'commit_msg_test_field': check_commit_msg_test_field,
     'cpplint': check_cpplint,
     'gofmt': check_gofmt,
     'google_java_format': check_google_java_format,
     'jsonlint': check_json,
-    'pylint': check_pylint,
+    'pylint': check_pylint2,
+    'pylint2': check_pylint2,
+    'pylint3': check_pylint3,
     'xmllint': check_xmllint,
 }
 
diff --git a/rh/hooks_unittest.py b/rh/hooks_unittest.py
index a1cb52e..e650038 100755
--- a/rh/hooks_unittest.py
+++ b/rh/hooks_unittest.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 # -*- coding:utf-8 -*-
 # Copyright 2016 The Android Open Source Project
 #
@@ -18,7 +18,6 @@
 
 from __future__ import print_function
 
-import mock
 import os
 import sys
 import unittest
@@ -28,9 +27,14 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# We have to import our local modules after the sys.path tweak.  We can't use
+# relative imports because this is an executable program, not a module.
+# pylint: disable=wrong-import-position
 import rh
-import rh.hooks
 import rh.config
+import rh.hooks
+from rh.sixish import mock
+from rh.sixish import string_types
 
 
 class HooksDocsTests(unittest.TestCase):
@@ -48,16 +52,18 @@ class HooksDocsTests(unittest.TestCase):
         """Extract the |section| text out of the readme."""
         ret = []
         in_section = False
-        for line in open(self.readme):
-            if not in_section:
-                # Look for the section like "## [Tool Paths]".
-                if line.startswith('#') and line.lstrip('#').strip() == section:
-                    in_section = True
-            else:
-                # Once we hit the next section (higher or lower), break.
-                if line[0] == '#':
-                    break
-                ret.append(line)
+        with open(self.readme) as fp:
+            for line in fp:
+                if not in_section:
+                    # Look for the section like "## [Tool Paths]".
+                    if (line.startswith('#') and
+                            line.lstrip('#').strip() == section):
+                        in_section = True
+                else:
+                    # Once we hit the next section (higher or lower), break.
+                    if line[0] == '#':
+                        break
+                    ret.append(line)
         return ''.join(ret)
 
     def testBuiltinHooks(self):
@@ -218,14 +224,14 @@ class UtilsTests(unittest.TestCase):
         # Just verify it returns something and doesn't crash.
         # pylint: disable=protected-access
         ret = rh.hooks._get_build_os_name()
-        self.assertTrue(isinstance(ret, str))
+        self.assertTrue(isinstance(ret, string_types))
         self.assertNotEqual(ret, '')
 
     def testGetHelperPath(self):
         """Check get_helper_path behavior."""
         # Just verify it doesn't crash.  It's a dirt simple func.
         ret = rh.hooks.get_helper_path('booga')
-        self.assertTrue(isinstance(ret, str))
+        self.assertTrue(isinstance(ret, string_types))
         self.assertNotEqual(ret, '')
 
 
@@ -240,16 +246,21 @@ class BuiltinHooksTests(unittest.TestCase):
                                   remote='remote')
         self.options = rh.hooks.HookOptions('hook name', [], {})
 
-    def _test_commit_messages(self, func, accept, msgs):
+    def _test_commit_messages(self, func, accept, msgs, files=None):
         """Helper for testing commit message hooks.
 
         Args:
           func: The hook function to test.
           accept: Whether all the |msgs| should be accepted.
           msgs: List of messages to test.
+          files: List of files to pass to the hook.
         """
+        if files:
+            diff = [rh.git.RawDiffEntry(file=x) for x in files]
+        else:
+            diff = []
         for desc in msgs:
-            ret = func(self.project, 'commit', desc, (), options=self.options)
+            ret = func(self.project, 'commit', desc, diff, options=self.options)
             if accept:
                 self.assertEqual(
                     ret, None, msg='Should have accepted: {{{%s}}}' % (desc,))
@@ -281,6 +292,20 @@ class BuiltinHooksTests(unittest.TestCase):
             self.assertIn('test_%s' % (hook,), dir(self),
                           msg='Missing unittest for builtin hook %s' % (hook,))
 
+    def test_bpfmt(self, mock_check, _mock_run):
+        """Verify the bpfmt builtin hook."""
+        # First call should do nothing as there are no files to check.
+        ret = rh.hooks.check_bpfmt(
+            self.project, 'commit', 'desc', (), options=self.options)
+        self.assertIsNone(ret)
+        self.assertFalse(mock_check.called)
+
+        # Second call will have some results.
+        diff = [rh.git.RawDiffEntry(file='Android.bp')]
+        ret = rh.hooks.check_bpfmt(
+            self.project, 'commit', 'desc', diff, options=self.options)
+        self.assertIsNotNone(ret)
+
     def test_checkpatch(self, mock_check, _mock_run):
         """Verify the checkpatch builtin hook."""
         ret = rh.hooks.check_checkpatch(
@@ -336,6 +361,8 @@ class BuiltinHooksTests(unittest.TestCase):
                 'subj',
                 'subj\n\nBUG=1234\n',
                 'subj\n\nBUG: 1234\n',
+                'subj\n\nBug: N/A\n',
+                'subj\n\nBug:\n',
             ))
 
     def test_commit_msg_changeid_field(self, _mock_check, _mock_run):
@@ -354,6 +381,112 @@ class BuiltinHooksTests(unittest.TestCase):
                 'subj\n\nChange-ID: I1234\n',
             ))
 
+    def test_commit_msg_prebuilt_apk_fields(self, _mock_check, _mock_run):
+        """Verify the check_commit_msg_prebuilt_apk_fields builtin hook."""
+        # Commits without APKs should pass.
+        self._test_commit_messages(
+            rh.hooks.check_commit_msg_prebuilt_apk_fields,
+            True,
+            (
+                'subj\nTest: test case\nBug: bug id\n',
+            ),
+            ['foo.cpp', 'bar.py',]
+        )
+
+        # Commits with APKs and all the required messages should pass.
+        self._test_commit_messages(
+            rh.hooks.check_commit_msg_prebuilt_apk_fields,
+            True,
+            (
+                ('Test App\n\nbar.apk\npackage: name=\'com.foo.bar\'\n'
+                 'versionCode=\'1001\'\nversionName=\'1.0.1001-A\'\n'
+                 'platformBuildVersionName=\'\'\ncompileSdkVersion=\'28\'\n'
+                 'compileSdkVersionCodename=\'9\'\nsdkVersion:\'16\'\n'
+                 'targetSdkVersion:\'28\'\n\nBuilt here:\n'
+                 'http://foo.bar.com/builder\n\n'
+                 'This build IS suitable for public release.\n\n'
+                 'Bug: 123\nTest: test\nChange-Id: XXXXXXX\n'),
+                ('Test App\n\nBuilt here:\nhttp://foo.bar.com/builder\n\n'
+                 'This build IS NOT suitable for public release.\n\n'
+                 'bar.apk\npackage: name=\'com.foo.bar\'\n'
+                 'versionCode=\'1001\'\nversionName=\'1.0.1001-A\'\n'
+                 'platformBuildVersionName=\'\'\ncompileSdkVersion=\'28\'\n'
+                 'compileSdkVersionCodename=\'9\'\nsdkVersion:\'16\'\n'
+                 'targetSdkVersion:\'28\'\n\nBug: 123\nTest: test\n'
+                 'Change-Id: XXXXXXX\n'),
+                ('Test App\n\nbar.apk\npackage: name=\'com.foo.bar\'\n'
+                 'versionCode=\'1001\'\nversionName=\'1.0.1001-A\'\n'
+                 'platformBuildVersionName=\'\'\ncompileSdkVersion=\'28\'\n'
+                 'compileSdkVersionCodename=\'9\'\nsdkVersion:\'16\'\n'
+                 'targetSdkVersion:\'28\'\n\nBuilt here:\n'
+                 'http://foo.bar.com/builder\n\n'
+                 'This build IS suitable for preview release but IS NOT '
+                 'suitable for public release.\n\n'
+                 'Bug: 123\nTest: test\nChange-Id: XXXXXXX\n'),
+                ('Test App\n\nbar.apk\npackage: name=\'com.foo.bar\'\n'
+                 'versionCode=\'1001\'\nversionName=\'1.0.1001-A\'\n'
+                 'platformBuildVersionName=\'\'\ncompileSdkVersion=\'28\'\n'
+                 'compileSdkVersionCodename=\'9\'\nsdkVersion:\'16\'\n'
+                 'targetSdkVersion:\'28\'\n\nBuilt here:\n'
+                 'http://foo.bar.com/builder\n\n'
+                 'This build IS NOT suitable for preview or public release.\n\n'
+                 'Bug: 123\nTest: test\nChange-Id: XXXXXXX\n'),
+            ),
+            ['foo.apk', 'bar.py',]
+        )
+
+        # Commits with APKs and without all the required messages should fail.
+        self._test_commit_messages(
+            rh.hooks.check_commit_msg_prebuilt_apk_fields,
+            False,
+            (
+                'subj\nTest: test case\nBug: bug id\n',
+                # Missing 'package'.
+                ('Test App\n\nbar.apk\n'
+                 'versionCode=\'1001\'\nversionName=\'1.0.1001-A\'\n'
+                 'platformBuildVersionName=\'\'\ncompileSdkVersion=\'28\'\n'
+                 'compileSdkVersionCodename=\'9\'\nsdkVersion:\'16\'\n'
+                 'targetSdkVersion:\'28\'\n\nBuilt here:\n'
+                 'http://foo.bar.com/builder\n\n'
+                 'This build IS suitable for public release.\n\n'
+                 'Bug: 123\nTest: test\nChange-Id: XXXXXXX\n'),
+                # Missing 'sdkVersion'.
+                ('Test App\n\nbar.apk\npackage: name=\'com.foo.bar\'\n'
+                 'versionCode=\'1001\'\nversionName=\'1.0.1001-A\'\n'
+                 'platformBuildVersionName=\'\'\ncompileSdkVersion=\'28\'\n'
+                 'compileSdkVersionCodename=\'9\'\n'
+                 'targetSdkVersion:\'28\'\n\nBuilt here:\n'
+                 'http://foo.bar.com/builder\n\n'
+                 'This build IS suitable for public release.\n\n'
+                 'Bug: 123\nTest: test\nChange-Id: XXXXXXX\n'),
+                # Missing 'targetSdkVersion'.
+                ('Test App\n\nbar.apk\npackage: name=\'com.foo.bar\'\n'
+                 'versionCode=\'1001\'\nversionName=\'1.0.1001-A\'\n'
+                 'platformBuildVersionName=\'\'\ncompileSdkVersion=\'28\'\n'
+                 'compileSdkVersionCodename=\'9\'\nsdkVersion:\'16\'\n'
+                 'Built here:\nhttp://foo.bar.com/builder\n\n'
+                 'This build IS suitable for public release.\n\n'
+                 'Bug: 123\nTest: test\nChange-Id: XXXXXXX\n'),
+                # Missing build location.
+                ('Test App\n\nbar.apk\npackage: name=\'com.foo.bar\'\n'
+                 'versionCode=\'1001\'\nversionName=\'1.0.1001-A\'\n'
+                 'platformBuildVersionName=\'\'\ncompileSdkVersion=\'28\'\n'
+                 'compileSdkVersionCodename=\'9\'\nsdkVersion:\'16\'\n'
+                 'targetSdkVersion:\'28\'\n\n'
+                 'This build IS suitable for public release.\n\n'
+                 'Bug: 123\nTest: test\nChange-Id: XXXXXXX\n'),
+                # Missing public release indication.
+                ('Test App\n\nbar.apk\npackage: name=\'com.foo.bar\'\n'
+                 'versionCode=\'1001\'\nversionName=\'1.0.1001-A\'\n'
+                 'platformBuildVersionName=\'\'\ncompileSdkVersion=\'28\'\n'
+                 'compileSdkVersionCodename=\'9\'\nsdkVersion:\'16\'\n'
+                 'targetSdkVersion:\'28\'\n\nBuilt here:\n'
+                 'http://foo.bar.com/builder\n\n'
+                 'Bug: 123\nTest: test\nChange-Id: XXXXXXX\n'),
+            ),
+            ['foo.apk', 'bar.py',]
+        )
+
     def test_commit_msg_test_field(self, _mock_check, _mock_run):
         """Verify the commit_msg_test_field builtin hook."""
         # Check some good messages.
@@ -406,7 +539,17 @@ class BuiltinHooksTests(unittest.TestCase):
 
     def test_pylint(self, mock_check, _mock_run):
         """Verify the pylint builtin hook."""
-        self._test_file_filter(mock_check, rh.hooks.check_pylint,
+        self._test_file_filter(mock_check, rh.hooks.check_pylint2,
+                               ('foo.py',))
+
+    def test_pylint2(self, mock_check, _mock_run):
+        """Verify the pylint2 builtin hook."""
+        self._test_file_filter(mock_check, rh.hooks.check_pylint2,
+                               ('foo.py',))
+
+    def test_pylint3(self, mock_check, _mock_run):
+        """Verify the pylint3 builtin hook."""
+        self._test_file_filter(mock_check, rh.hooks.check_pylint3,
                                ('foo.py',))
 
     def test_xmllint(self, mock_check, _mock_run):
@@ -414,6 +557,20 @@ class BuiltinHooksTests(unittest.TestCase):
         self._test_file_filter(mock_check, rh.hooks.check_xmllint,
                                ('foo.xml',))
 
+    def test_android_test_mapping_format(self, mock_check, _mock_run):
+        """Verify the android_test_mapping_format builtin hook."""
+        # First call should do nothing as there are no files to check.
+        ret = rh.hooks.check_android_test_mapping(
+            self.project, 'commit', 'desc', (), options=self.options)
+        self.assertIsNone(ret)
+        self.assertFalse(mock_check.called)
+
+        # Second call will have some results.
+        diff = [rh.git.RawDiffEntry(file='TEST_MAPPING')]
+        ret = rh.hooks.check_android_test_mapping(
+            self.project, 'commit', 'desc', diff, options=self.options)
+        self.assertIsNotNone(ret)
+
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/rh/results.py b/rh/results.py
index 8fa956f..062aaa3 100644
--- a/rh/results.py
+++ b/rh/results.py
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Common errors thrown when repo presubmit checks fail."""
+"""Common errors thrown when repo preupload checks fail."""
 
 from __future__ import print_function
 
@@ -54,10 +54,14 @@ class HookResult(object):
     def __bool__(self):
         return bool(self.error)
 
+    # pylint: disable=nonzero-method
     def __nonzero__(self):
         """Python 2/3 glue."""
         return self.__bool__()
 
+    def is_warning(self):
+        return False
+
 
 class HookCommandResult(HookResult):
     """A single hook result based on a CommandResult."""
@@ -71,3 +75,6 @@ class HookCommandResult(HookResult):
 
     def __bool__(self):
         return self.result.returncode not in (None, 0)
+
+    def is_warning(self):
+        return self.result.returncode == 77
diff --git a/rh/shell.py b/rh/shell.py
index 6c943d3..f466f63 100644
--- a/rh/shell.py
+++ b/rh/shell.py
@@ -25,6 +25,9 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# pylint: disable=wrong-import-position
+from rh.sixish import string_types
+
 
 # For use by ShellQuote.  Match all characters that the shell might treat
 # specially.  This means a number of things:
@@ -68,28 +71,26 @@ def shell_quote(s):
     Returns:
       A safely (possibly quoted) string.
     """
-    s = s.encode('utf-8')
+    if isinstance(s, bytes):
+        s = s.encode('utf-8')
 
     # See if no quoting is needed so we can return the string as-is.
     for c in s:
         if c in _SHELL_QUOTABLE_CHARS:
             break
     else:
-        if not s:
-            return "''"
-        else:
-            return s
+        return s if s else u"''"
 
     # See if we can use single quotes first.  Output is nicer.
     if "'" not in s:
-        return "'%s'" % s
+        return u"'%s'" % s
 
     # Have to use double quotes.  Escape the few chars that still expand when
     # used inside of double quotes.
     for c in _SHELL_ESCAPE_CHARS:
         if c in s:
             s = s.replace(c, r'\%s' % c)
-    return '"%s"' % s
+    return u'"%s"' % s
 
 
 def shell_unquote(s):
@@ -155,11 +156,11 @@ def boolean_shell_value(sval, default):
     if sval is None:
         return default
 
-    if isinstance(sval, basestring):
+    if isinstance(sval, string_types):
         s = sval.lower()
         if s in ('yes', 'y', '1', 'true'):
             return True
-        elif s in ('no', 'n', '0', 'false'):
+        if s in ('no', 'n', '0', 'false'):
             return False
 
     raise ValueError('Could not decode as a boolean value: %r' % (sval,))
diff --git a/rh/shell_unittest.py b/rh/shell_unittest.py
index bd98ec1..47182a5 100755
--- a/rh/shell_unittest.py
+++ b/rh/shell_unittest.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 # -*- coding:utf-8 -*-
 # Copyright 2016 The Android Open Source Project
 #
@@ -28,6 +28,9 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# We have to import our local modules after the sys.path tweak.  We can't use
+# relative imports because this is an executable program, not a module.
+# pylint: disable=wrong-import-position
 import rh.shell
 
 
@@ -46,7 +49,7 @@ class DiffTestCase(unittest.TestCase):
 
     def _testData(self, functor, tests, check_type=True):
         """Process a dict of test data."""
-        for test_output, test_input in tests.iteritems():
+        for test_output, test_input in tests.items():
             result = functor(test_input)
             self._assertEqual(functor.__name__, test_input, test_output, result)
 
@@ -65,8 +68,8 @@ class ShellQuoteTest(DiffTestCase):
         # Dict of expected output strings to input lists.
         tests_quote = {
             "''": '',
-            'a': unicode('a'),
-            "'a b c'": unicode('a b c'),
+            'a': u'a',
+            "'a b c'": u'a b c',
             "'a\tb'": 'a\tb',
             "'/a$file'": '/a$file',
             "'/a#file'": '/a#file',
@@ -92,7 +95,7 @@ class ShellQuoteTest(DiffTestCase):
 
         # Test that the operations are reversible.
         self._testData(aux, {k: k for k in tests_quote.values()}, False)
-        self._testData(aux, {k: k for k in tests_quote.keys()}, False)
+        self._testData(aux, {k: k for k in tests_quote}, False)
 
 
 class CmdToStrTest(DiffTestCase):
@@ -105,7 +108,7 @@ class CmdToStrTest(DiffTestCase):
             r"'a b' c": ['a b', 'c'],
             r'''a "b'c"''': ['a', "b'c"],
             r'''a "/'\$b" 'a b c' "xy'z"''':
-                [unicode('a'), "/'$b", 'a b c', "xy'z"],
+                [u'a', "/'$b", 'a b c', "xy'z"],
             '': [],
         }
         self._testData(rh.shell.cmd_to_str, tests)
diff --git a/rh/signals.py b/rh/signals.py
index 5d68bf1..d6e4cd9 100644
--- a/rh/signals.py
+++ b/rh/signals.py
@@ -41,7 +41,7 @@ def relay_signal(handler, signum, frame):
     """
     if handler in (None, signal.SIG_IGN):
         return True
-    elif handler == signal.SIG_DFL:
+    if handler == signal.SIG_DFL:
         # This scenario is a fairly painful to handle fully, thus we just
         # state we couldn't handle it and leave it to client code.
         return False
diff --git a/rh/sixish.py b/rh/sixish.py
new file mode 100644
index 0000000..693598c
--- /dev/null
+++ b/rh/sixish.py
@@ -0,0 +1,69 @@
+# -*- coding:utf-8 -*-
+# Copyright 2017 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Local version of the standard six module.
+
+Since this module is often unavailable by default on distros (or only available
+for specific versions), manage our own little copy.
+"""
+
+from __future__ import print_function
+
+import os
+import sys
+
+_path = os.path.realpath(__file__ + '/../..')
+if sys.path[0] != _path:
+    sys.path.insert(0, _path)
+del _path
+
+
+# Our attempts to wrap things below for diff versions of python confuse pylint.
+# pylint: disable=import-error,no-name-in-module,unused-import
+
+
+try:
+    import configparser
+except ImportError:
+    import ConfigParser as configparser
+
+
+# We allow mock to be disabled as it's not needed by non-unittest code.
+try:
+    import unittest.mock as mock
+except ImportError:
+    try:
+        import mock
+    except ImportError:
+        pass
+
+
+if sys.version_info.major < 3:
+    # pylint: disable=basestring-builtin,undefined-variable
+    string_types = basestring
+else:
+    string_types = str
+
+
+def setenv(var, val):
+    """Set |var| in the environment to |val|.
+
+    Python 2 wants ASCII strings, not unicode.
+    Python 3 only wants unicode strings.
+    """
+    try:
+        os.environ[var] = val
+    except UnicodeEncodeError:
+        os.environ[var] = val.encode('utf-8')
diff --git a/rh/terminal.py b/rh/terminal.py
index 9eda0ec..c549f12 100644
--- a/rh/terminal.py
+++ b/rh/terminal.py
@@ -28,6 +28,7 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# pylint: disable=wrong-import-position
 import rh.shell
 
 
@@ -141,11 +142,12 @@ def print_status_line(line, print_newline=False):
 def get_input(prompt):
     """Python 2/3 glue for raw_input/input differences."""
     try:
+        # pylint: disable=raw_input-builtin
         return raw_input(prompt)
     except NameError:
         # Python 3 renamed raw_input() to input(), which is safe to call since
         # it does not evaluate the input.
-        # pylint: disable=bad-builtin
+        # pylint: disable=bad-builtin,input-builtin
         return input(prompt)
 
 
diff --git a/rh/utils.py b/rh/utils.py
index bec081c..b3ed338 100644
--- a/rh/utils.py
+++ b/rh/utils.py
@@ -32,8 +32,10 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# pylint: disable=wrong-import-position
 import rh.shell
 import rh.signals
+from rh.sixish import string_types
 
 
 class CommandResult(object):
@@ -86,12 +88,10 @@ class RunCommandError(Exception):
         return '\n'.join(items)
 
     def __str__(self):
-        # __str__ needs to return ascii, thus force a conversion to be safe.
-        return self.stringify().decode('utf-8', 'replace').encode(
-            'ascii', 'xmlcharrefreplace')
+        return self.stringify()
 
     def __eq__(self, other):
-        return (type(self) == type(other) and
+        return (isinstance(other, type(self)) and
                 self.args == other.args)
 
     def __ne__(self, other):
@@ -130,6 +130,10 @@ def sudo_run_command(cmd, user='root', **kwargs):
       Barring that, see RunCommand's documentation- it can raise the same things
       RunCommand does.
     """
+    # We don't use this anywhere, so it's easier to not bother supporting it.
+    assert not isinstance(cmd, string_types), 'shell commands not supported'
+    assert 'shell' not in kwargs, 'shell=True is not supported'
+
     sudo_cmd = ['sudo']
 
     if user == 'root' and os.geteuid() == 0:
@@ -143,22 +147,12 @@ def sudo_run_command(cmd, user='root', **kwargs):
     extra_env = kwargs.pop('extra_env', None)
     extra_env = {} if extra_env is None else extra_env.copy()
 
-    sudo_cmd.extend('%s=%s' % (k, v) for k, v in extra_env.iteritems())
+    sudo_cmd.extend('%s=%s' % (k, v) for k, v in extra_env.items())
 
     # Finally, block people from passing options to sudo.
     sudo_cmd.append('--')
 
-    if isinstance(cmd, basestring):
-        # We need to handle shell ourselves so the order is correct:
-        #  $ sudo [sudo args] -- bash -c '[shell command]'
-        # If we let RunCommand take care of it, we'd end up with:
-        #  $ bash -c 'sudo [sudo args] -- [shell command]'
-        shell = kwargs.pop('shell', False)
-        if not shell:
-            raise Exception('Cannot run a string command without a shell')
-        sudo_cmd.extend(['/bin/bash', '-c', cmd])
-    else:
-        sudo_cmd.extend(cmd)
+    sudo_cmd.extend(cmd)
 
     return run_command(sudo_cmd, **kwargs)
 
@@ -221,6 +215,7 @@ class _Popen(subprocess.Popen):
     process has knowingly been waitpid'd already.
     """
 
+    # pylint: disable=arguments-differ
     def send_signal(self, signum):
         if self.returncode is not None:
             # The original implementation in Popen allows signaling whatever
@@ -257,7 +252,8 @@ class _Popen(subprocess.Popen):
                 raise
 
 
-# pylint: disable=redefined-builtin
+# We use the keyword arg |input| which trips up pylint checks.
+# pylint: disable=redefined-builtin,input-builtin
 def run_command(cmd, error_message=None, redirect_stdout=False,
                 redirect_stderr=False, cwd=None, input=None,
                 shell=False, env=None, extra_env=None, ignore_sigint=False,
@@ -328,8 +324,13 @@ def run_command(cmd, error_message=None, redirect_stdout=False,
     kill_timeout = float(kill_timeout)
 
     def _get_tempfile():
+        kwargs = {}
+        if sys.version_info.major < 3:
+            kwargs['bufsize'] = 0
+        else:
+            kwargs['buffering'] = 0
         try:
-            return tempfile.TemporaryFile(bufsize=0)
+            return tempfile.TemporaryFile(**kwargs)
         except EnvironmentError as e:
             if e.errno != errno.ENOENT:
                 raise
@@ -338,12 +339,14 @@ def run_command(cmd, error_message=None, redirect_stdout=False,
             # issue in this particular case since our usage gurantees deletion,
             # and since this is primarily triggered during hard cgroups
             # shutdown.
-            return tempfile.TemporaryFile(bufsize=0, dir='/tmp')
+            return tempfile.TemporaryFile(dir='/tmp', **kwargs)
 
     # Modify defaults based on parameters.
     # Note that tempfiles must be unbuffered else attempts to read
     # what a separate process did to that file can result in a bad
     # view of the file.
+    # The Popen API accepts either an int or a file handle for stdout/stderr.
+    # pylint: disable=redefined-variable-type
     if log_stdout_to_file:
         stdout = open(log_stdout_to_file, 'w+')
     elif stdout_to_pipe:
@@ -355,6 +358,7 @@ def run_command(cmd, error_message=None, redirect_stdout=False,
         stderr = subprocess.STDOUT
     elif redirect_stderr:
         stderr = _get_tempfile()
+    # pylint: enable=redefined-variable-type
 
     # If subprocesses have direct access to stdout or stderr, they can bypass
     # our buffers, so we need to flush to ensure that output is not interleaved.
@@ -364,13 +368,14 @@ def run_command(cmd, error_message=None, redirect_stdout=False,
 
     # If input is a string, we'll create a pipe and send it through that.
     # Otherwise we assume it's a file object that can be read from directly.
-    if isinstance(input, basestring):
+    if isinstance(input, string_types):
         stdin = subprocess.PIPE
+        input = input.encode('utf-8')
     elif input is not None:
         stdin = input
         input = None
 
-    if isinstance(cmd, basestring):
+    if isinstance(cmd, string_types):
         if not shell:
             raise Exception('Cannot run a string command without a shell')
         cmd = ['/bin/bash', '-c', cmd]
@@ -386,36 +391,30 @@ def run_command(cmd, error_message=None, redirect_stdout=False,
     cmd_result.cmd = cmd
 
     proc = None
-    # Verify that the signals modules is actually usable, and won't segfault
-    # upon invocation of getsignal.  See signals.SignalModuleUsable for the
-    # details and upstream python bug.
-    use_signals = rh.signals.signal_module_usable()
     try:
         proc = _Popen(cmd, cwd=cwd, stdin=stdin, stdout=stdout,
                       stderr=stderr, shell=False, env=env,
                       close_fds=close_fds)
 
-        if use_signals:
-            old_sigint = signal.getsignal(signal.SIGINT)
-            if ignore_sigint:
-                handler = signal.SIG_IGN
-            else:
-                handler = functools.partial(
-                    _kill_child_process, proc, int_timeout, kill_timeout, cmd,
-                    old_sigint)
-            signal.signal(signal.SIGINT, handler)
+        old_sigint = signal.getsignal(signal.SIGINT)
+        if ignore_sigint:
+            handler = signal.SIG_IGN
+        else:
+            handler = functools.partial(
+                _kill_child_process, proc, int_timeout, kill_timeout, cmd,
+                old_sigint)
+        signal.signal(signal.SIGINT, handler)
 
-            old_sigterm = signal.getsignal(signal.SIGTERM)
-            handler = functools.partial(_kill_child_process, proc, int_timeout,
-                                        kill_timeout, cmd, old_sigterm)
-            signal.signal(signal.SIGTERM, handler)
+        old_sigterm = signal.getsignal(signal.SIGTERM)
+        handler = functools.partial(_kill_child_process, proc, int_timeout,
+                                    kill_timeout, cmd, old_sigterm)
+        signal.signal(signal.SIGTERM, handler)
 
         try:
             (cmd_result.output, cmd_result.error) = proc.communicate(input)
         finally:
-            if use_signals:
-                signal.signal(signal.SIGINT, old_sigint)
-                signal.signal(signal.SIGTERM, old_sigterm)
+            signal.signal(signal.SIGINT, old_sigint)
+            signal.signal(signal.SIGTERM, old_sigterm)
 
             if stdout and not log_stdout_to_file and not stdout_to_pipe:
                 # The linter is confused by how stdout is a file & an int.
@@ -451,66 +450,16 @@ def run_command(cmd, error_message=None, redirect_stdout=False,
     finally:
         if proc is not None:
             # Ensure the process is dead.
+            # Some pylint3 versions are confused here.
+            # pylint: disable=too-many-function-args
             _kill_child_process(proc, int_timeout, kill_timeout, cmd, None,
                                 None, None)
 
-    return cmd_result
-# pylint: enable=redefined-builtin
+    # Make sure output is returned as a string rather than bytes.
+    if cmd_result.output is not None:
+        cmd_result.output = cmd_result.output.decode('utf-8', 'replace')
+    if cmd_result.error is not None:
+        cmd_result.error = cmd_result.error.decode('utf-8', 'replace')
 
-
-def collection(classname, **kwargs):
-    """Create a new class with mutable named members.
-
-    This is like collections.namedtuple, but mutable.  Also similar to the
-    python 3.3 types.SimpleNamespace.
-
-    Example:
-      # Declare default values for this new class.
-      Foo = collection('Foo', a=0, b=10)
-      # Create a new class but set b to 4.
-      foo = Foo(b=4)
-      # Print out a (will be the default 0) and b (will be 4).
-      print('a = %i, b = %i' % (foo.a, foo.b))
-    """
-
-    def sn_init(self, **kwargs):
-        """The new class's __init__ function."""
-        # First verify the kwargs don't have excess settings.
-        valid_keys = set(self.__slots__[1:])
-        these_keys = set(kwargs.keys())
-        invalid_keys = these_keys - valid_keys
-        if invalid_keys:
-            raise TypeError('invalid keyword arguments for this object: %r' %
-                            invalid_keys)
-
-        # Now initialize this object.
-        for k in valid_keys:
-            setattr(self, k, kwargs.get(k, self.__defaults__[k]))
-
-    def sn_repr(self):
-        """The new class's __repr__ function."""
-        return '%s(%s)' % (classname, ', '.join(
-            '%s=%r' % (k, getattr(self, k)) for k in self.__slots__[1:]))
-
-    # Give the new class a unique name and then generate the code for it.
-    classname = 'Collection_%s' % classname
-    expr = '\n'.join((
-        'class %(classname)s(object):',
-        '  __slots__ = ["__defaults__", "%(slots)s"]',
-        '  __defaults__ = {}',
-    )) % {
-        'classname': classname,
-        'slots': '", "'.join(sorted(str(k) for k in kwargs)),
-    }
-
-    # Create the class in a local namespace as exec requires.
-    namespace = {}
-    exec expr in namespace  # pylint: disable=exec-used
-    new_class = namespace[classname]
-
-    # Bind the helpers.
-    new_class.__defaults__ = kwargs.copy()
-    new_class.__init__ = sn_init
-    new_class.__repr__ = sn_repr
-
-    return new_class
+    return cmd_result
+# pylint: enable=redefined-builtin,input-builtin
diff --git a/rh/utils_unittest.py b/rh/utils_unittest.py
new file mode 100755
index 0000000..881d031
--- /dev/null
+++ b/rh/utils_unittest.py
@@ -0,0 +1,214 @@
+#!/usr/bin/env python3
+# -*- coding:utf-8 -*-
+# Copyright 2019 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unittests for the utils module."""
+
+from __future__ import print_function
+
+import os
+import sys
+import unittest
+
+_path = os.path.realpath(__file__ + '/../..')
+if sys.path[0] != _path:
+    sys.path.insert(0, _path)
+del _path
+
+# We have to import our local modules after the sys.path tweak.  We can't use
+# relative imports because this is an executable program, not a module.
+# pylint: disable=wrong-import-position
+import rh
+import rh.utils
+from rh.sixish import mock
+
+
+class CommandResultTests(unittest.TestCase):
+    """Verify behavior of CommandResult object."""
+
+    def test_empty_cmdstr(self):
+        """Check cmdstr with an empty command."""
+        result = rh.utils.CommandResult(cmd=[])
+        self.assertEqual('', result.cmdstr)
+
+    def test_basic_cmdstr(self):
+        """Check cmdstr with a basic command command."""
+        result = rh.utils.CommandResult(cmd=['ls', 'a b'])
+        self.assertEqual("ls 'a b'", result.cmdstr)
+
+    def test_str(self):
+        """Check str() handling."""
+        # We don't enforce much, just that it doesn't crash.
+        result = rh.utils.CommandResult()
+        self.assertNotEqual('', str(result))
+        result = rh.utils.CommandResult(cmd=[])
+        self.assertNotEqual('', str(result))
+
+    def test_repr(self):
+        """Check repr() handling."""
+        # We don't enforce much, just that it doesn't crash.
+        result = rh.utils.CommandResult()
+        self.assertNotEqual('', repr(result))
+        result = rh.utils.CommandResult(cmd=[])
+        self.assertNotEqual('', repr(result))
+
+
+class RunCommandErrorTests(unittest.TestCase):
+    """Verify behavior of RunCommandError object."""
+
+    def setUp(self):
+        self.result = rh.utils.CommandResult(cmd=['mycmd'])
+
+    def test_basic(self):
+        """Basic test we can create a normal instance."""
+        rh.utils.RunCommandError('msg', self.result)
+        rh.utils.RunCommandError('msg', self.result, exception=Exception('bad'))
+
+    def test_stringify(self):
+        """Check stringify() handling."""
+        # We don't assert much so we leave flexibility in changing format.
+        err = rh.utils.RunCommandError('msg', self.result)
+        self.assertIn('mycmd', err.stringify())
+        err = rh.utils.RunCommandError('msg', self.result,
+                                       exception=Exception('bad'))
+        self.assertIn('mycmd', err.stringify())
+
+    def test_str(self):
+        """Check str() handling."""
+        # We don't assert much so we leave flexibility in changing format.
+        err = rh.utils.RunCommandError('msg', self.result)
+        self.assertIn('mycmd', str(err))
+        err = rh.utils.RunCommandError('msg', self.result,
+                                       exception=Exception('bad'))
+        self.assertIn('mycmd', str(err))
+
+    def test_repr(self):
+        """Check repr() handling."""
+        # We don't assert much so we leave flexibility in changing format.
+        err = rh.utils.RunCommandError('msg', self.result)
+        self.assertNotEqual('', repr(err))
+        err = rh.utils.RunCommandError('msg', self.result,
+                                       exception=Exception('bad'))
+        self.assertNotEqual('', repr(err))
+
+    def test_eq(self):
+        """Check object equality."""
+        # Note: We explicitly do not use assertEqual here.
+        # We also explicitly compare to ourselves to make sure our __eq__ works.
+        # We also disable bad-option-value because this is a pylint3 warning.
+        # pylint: disable=bad-option-value,comparison-with-itself
+        err1 = rh.utils.RunCommandError('msg', self.result)
+        self.assertTrue(err1 == err1)
+        err2 = rh.utils.RunCommandError('msg', self.result)
+        self.assertTrue(err1 == err2)
+        err3 = rh.utils.RunCommandError('foo', self.result)
+        self.assertFalse(err1 == err3)
+
+    def test_ne(self):
+        """Check object inequality."""
+        # Note: We explicitly do not use assertNotEqual here.
+        # We also explicitly compare to ourselves to make sure our __eq__ works.
+        # We also disable bad-option-value because this is a pylint3 warning.
+        # pylint: disable=bad-option-value,comparison-with-itself
+        err1 = rh.utils.RunCommandError('msg', self.result)
+        self.assertFalse(err1 != err1)
+        err2 = rh.utils.RunCommandError('msg', self.result)
+        self.assertFalse(err1 != err2)
+        err3 = rh.utils.RunCommandError('foo', self.result)
+        self.assertTrue(err1 != err3)
+
+
+# We shouldn't require sudo to run unittests :).
+@mock.patch.object(rh.utils, 'run_command')
+@mock.patch.object(os, 'geteuid', return_value=1000)
+class SudoRunCommandTests(unittest.TestCase):
+    """Verify behavior of sudo_run_command helper."""
+
+    def test_run_as_root_as_root(self, mock_geteuid, mock_run):
+        """Check behavior when we're already root."""
+        mock_geteuid.return_value = 0
+        ret = rh.utils.sudo_run_command(['ls'], user='root')
+        self.assertIsNotNone(ret)
+        args, _kwargs = mock_run.call_args
+        self.assertEqual((['ls'],), args)
+
+    def test_run_as_root_as_nonroot(self, _mock_geteuid, mock_run):
+        """Check behavior when we're not already root."""
+        ret = rh.utils.sudo_run_command(['ls'], user='root')
+        self.assertIsNotNone(ret)
+        args, _kwargs = mock_run.call_args
+        self.assertEqual((['sudo', '--', 'ls'],), args)
+
+    def test_run_as_nonroot_as_nonroot(self, _mock_geteuid, mock_run):
+        """Check behavior when we're not already root."""
+        ret = rh.utils.sudo_run_command(['ls'], user='nobody')
+        self.assertIsNotNone(ret)
+        args, _kwargs = mock_run.call_args
+        self.assertEqual((['sudo', '-u', 'nobody', '--', 'ls'],), args)
+
+    def test_env(self, _mock_geteuid, mock_run):
+        """Check passing through env vars."""
+        ret = rh.utils.sudo_run_command(['ls'], extra_env={'FOO': 'bar'})
+        self.assertIsNotNone(ret)
+        args, _kwargs = mock_run.call_args
+        self.assertEqual((['sudo', 'FOO=bar', '--', 'ls'],), args)
+
+    def test_shell(self, _mock_geteuid, _mock_run):
+        """Check attempts to use shell code are rejected."""
+        with self.assertRaises(AssertionError):
+            rh.utils.sudo_run_command('foo')
+        with self.assertRaises(AssertionError):
+            rh.utils.sudo_run_command(['ls'], shell=True)
+
+
+class RunCommandTests(unittest.TestCase):
+    """Verify behavior of run_command helper."""
+
+    def test_basic(self):
+        """Simple basic test."""
+        ret = rh.utils.run_command(['true'])
+        self.assertEqual('true', ret.cmdstr)
+        self.assertIsNone(ret.output)
+        self.assertIsNone(ret.error)
+
+    def test_stdout_capture(self):
+        """Verify output capturing works."""
+        ret = rh.utils.run_command(['echo', 'hi'], redirect_stdout=True)
+        self.assertEqual('hi\n', ret.output)
+        self.assertIsNone(ret.error)
+
+    def test_stderr_capture(self):
+        """Verify stderr capturing works."""
+        ret = rh.utils.run_command(['sh', '-c', 'echo hi >&2'],
+                                   redirect_stderr=True)
+        self.assertIsNone(ret.output)
+        self.assertEqual('hi\n', ret.error)
+
+    def test_stdout_utf8(self):
+        """Verify reading UTF-8 data works."""
+        ret = rh.utils.run_command(['printf', r'\xc3\x9f'],
+                                   redirect_stdout=True)
+        self.assertEqual(u'ß', ret.output)
+        self.assertIsNone(ret.error)
+
+    def test_stdin_utf8(self):
+        """Verify writing UTF-8 data works."""
+        ret = rh.utils.run_command(['cat'], redirect_stdout=True, input=u'ß')
+        self.assertEqual(u'ß', ret.output)
+        self.assertIsNone(ret.error)
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/tools/android_test_mapping_format.py b/tools/android_test_mapping_format.py
new file mode 100755
index 0000000..14b02f5
--- /dev/null
+++ b/tools/android_test_mapping_format.py
@@ -0,0 +1,204 @@
+#!/usr/bin/python
+# -*- coding:utf-8 -*-
+# Copyright 2018 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Validate TEST_MAPPING files in Android source code.
+
+The goal of this script is to validate the format of TEST_MAPPING files:
+1. It must be a valid json file.
+2. Each test group must have a list of test that containing name and options.
+3. Each import must have only one key `path` and one value for the path to
+   import TEST_MAPPING files.
+"""
+
+from __future__ import print_function
+
+import argparse
+import json
+import os
+import re
+import sys
+
+_path = os.path.realpath(__file__ + '/../..')
+if sys.path[0] != _path:
+    sys.path.insert(0, _path)
+del _path
+
+# We have to import our local modules after the sys.path tweak.  We can't use
+# relative imports because this is an executable program, not a module.
+# pylint: disable=wrong-import-position
+import rh.git
+
+IMPORTS = 'imports'
+NAME = 'name'
+OPTIONS = 'options'
+PATH = 'path'
+HOST = 'host'
+PREFERRED_TARGETS = 'preferred_targets'
+FILE_PATTERNS = 'file_patterns'
+TEST_MAPPING_URL = (
+    'https://source.android.com/compatibility/tests/development/'
+    'test-mapping')
+
+# Pattern used to identify line-level '//'-format comment in TEST_MAPPING file.
+_COMMENTS_RE = re.compile(r'^\s*//')
+
+
+if sys.version_info.major < 3:
+    # pylint: disable=basestring-builtin,undefined-variable
+    string_types = basestring
+else:
+    string_types = str
+
+
+class Error(Exception):
+    """Base exception for all custom exceptions in this module."""
+
+
+class InvalidTestMappingError(Error):
+    """Exception to raise when detecting an invalid TEST_MAPPING file."""
+
+
+def filter_comments(json_data):
+    """Remove '//'-format comments in TEST_MAPPING file to valid format.
+
+    Args:
+        json_data: TEST_MAPPING file content (as a string).
+
+    Returns:
+        Valid json string without comments.
+    """
+    return ''.join('\n' if _COMMENTS_RE.match(x) else x for x in
+                   json_data.splitlines())
+
+
+def _validate_import(entry, test_mapping_file):
+    """Validate an import setting.
+
+    Args:
+        entry: A dictionary of an import setting.
+        test_mapping_file: Path to the TEST_MAPPING file to be validated.
+
+    Raises:
+        InvalidTestMappingError: if the import setting is invalid.
+    """
+    if len(entry) != 1:
+        raise InvalidTestMappingError(
+            'Invalid import config in test mapping file %s. each import can '
+            'only have one `path` setting. Failed entry: %s' %
+            (test_mapping_file, entry))
+    if list(entry.keys())[0] != PATH:
+        raise InvalidTestMappingError(
+            'Invalid import config in test mapping file %s. import can only '
+            'have one `path` setting. Failed entry: %s' %
+            (test_mapping_file, entry))
+
+
+def _validate_test(test, test_mapping_file):
+    """Validate a test declaration.
+
+    Args:
+        entry: A dictionary of a test declaration.
+        test_mapping_file: Path to the TEST_MAPPING file to be validated.
+
+    Raises:
+        InvalidTestMappingError: if the a test declaration is invalid.
+    """
+    if NAME not in test:
+        raise InvalidTestMappingError(
+            'Invalid test config in test mapping file %s. test config must '
+            'a `name` setting. Failed test config: %s' %
+            (test_mapping_file, test))
+    if not isinstance(test.get(HOST, False), bool):
+        raise InvalidTestMappingError(
+            'Invalid test config in test mapping file %s. `host` setting in '
+            'test config can only have boolean value of `true` or `false`. '
+            'Failed test config: %s' % (test_mapping_file, test))
+    preferred_targets = test.get(PREFERRED_TARGETS, [])
+    if (not isinstance(preferred_targets, list) or
+            any(not isinstance(t, string_types) for t in preferred_targets)):
+        raise InvalidTestMappingError(
+            'Invalid test config in test mapping file %s. `preferred_targets` '
+            'setting in test config can only be a list of strings. Failed test '
+            'config: %s' % (test_mapping_file, test))
+    file_patterns = test.get(FILE_PATTERNS, [])
+    if (not isinstance(file_patterns, list) or
+            any(not isinstance(p, string_types) for p in file_patterns)):
+        raise InvalidTestMappingError(
+            'Invalid test config in test mapping file %s. `file_patterns` '
+            'setting in test config can only be a list of strings. Failed test '
+            'config: %s' % (test_mapping_file, test))
+    for option in test.get(OPTIONS, []):
+        if len(option) != 1:
+            raise InvalidTestMappingError(
+                'Invalid option setting in test mapping file %s. each option '
+                'setting can only have one key-val setting. Failed entry: %s' %
+                (test_mapping_file, option))
+
+
+def _load_file(test_mapping_file):
+    """Load a TEST_MAPPING file as a json file."""
+    try:
+        return json.loads(filter_comments(test_mapping_file))
+    except ValueError as e:
+        # The file is not a valid JSON file.
+        print(
+            'Failed to parse JSON file %s, error: %s' % (test_mapping_file, e),
+            file=sys.stderr)
+        raise
+
+
+def process_file(test_mapping_file):
+    """Validate a TEST_MAPPING file."""
+    test_mapping = _load_file(test_mapping_file)
+    # Validate imports.
+    for import_entry in test_mapping.get(IMPORTS, []):
+        _validate_import(import_entry, test_mapping_file)
+    # Validate tests.
+    all_tests = [test for group, tests in test_mapping.items()
+                 if group != IMPORTS for test in tests]
+    for test in all_tests:
+        _validate_test(test, test_mapping_file)
+
+
+def get_parser():
+    """Return a command line parser."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--commit', type=str,
+                        help='Specify the commit to validate.')
+    parser.add_argument('project_dir')
+    parser.add_argument('files', nargs='+')
+    return parser
+
+
+def main(argv):
+    parser = get_parser()
+    opts = parser.parse_args(argv)
+    try:
+        for filename in opts.files:
+            if opts.commit:
+                json_data = rh.git.get_file_content(opts.commit, filename)
+            else:
+                with open(os.path.join(opts.project_dir, filename)) as f:
+                    json_data = f.read()
+            process_file(json_data)
+    except:
+        print('Visit %s for details about the format of TEST_MAPPING '
+              'file.' % TEST_MAPPING_URL, file=sys.stderr)
+        raise
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[1:]))
diff --git a/tools/android_test_mapping_format_unittest.py b/tools/android_test_mapping_format_unittest.py
new file mode 100755
index 0000000..9191a25
--- /dev/null
+++ b/tools/android_test_mapping_format_unittest.py
@@ -0,0 +1,309 @@
+#!/usr/bin/env python3
+# -*- coding:utf-8 -*-
+# Copyright 2018 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import shutil
+import tempfile
+import unittest
+
+import android_test_mapping_format
+
+
+VALID_TEST_MAPPING = r"""
+{
+  "presubmit": [
+    {
+      "name": "CtsWindowManagerDeviceTestCases",
+      "options": [
+        {
+          "include-annotation": "android.platform.test.annotations.Presubmit"
+        }
+      ]
+    }
+  ],
+  "postsubmit": [
+    {
+      "name": "CtsWindowManagerDeviceTestCases",
+      "host": true,
+      "preferred_targets": ["a", "b"],
+      "file_patterns": [".*\\.java"]
+    }
+  ],
+  "imports": [
+    {
+      "path": "frameworks/base/services/core/java/com/android/server/am"
+    },
+    {
+      "path": "frameworks/base/services/core/java/com/android/server/wm"
+    }
+  ]
+}
+"""
+
+BAD_JSON = """
+{wrong format}
+"""
+
+BAD_TEST_WRONG_KEY = """
+{
+  "presubmit": [
+    {
+      "bad_name": "CtsWindowManagerDeviceTestCases"
+    }
+  ]
+}
+"""
+
+BAD_TEST_WRONG_HOST_VALUE = """
+{
+  "presubmit": [
+    {
+      "name": "CtsWindowManagerDeviceTestCases",
+      "host": "bad_value"
+    }
+  ]
+}
+"""
+
+
+BAD_TEST_WRONG_PREFERRED_TARGETS_VALUE_NONE_LIST = """
+{
+  "presubmit": [
+    {
+      "name": "CtsWindowManagerDeviceTestCases",
+      "preferred_targets": "bad_value"
+    }
+  ]
+}
+"""
+
+BAD_TEST_WRONG_PREFERRED_TARGETS_VALUE_WRONG_TYPE = """
+{
+  "presubmit": [
+    {
+      "name": "CtsWindowManagerDeviceTestCases",
+      "preferred_targets": ["bad_value", 123]
+    }
+  ]
+}
+"""
+
+BAD_TEST_WRONG_OPTION = """
+{
+  "presubmit": [
+    {
+      "name": "CtsWindowManagerDeviceTestCases",
+      "options": [
+        {
+          "include-annotation": "android.platform.test.annotations.Presubmit",
+          "bad_option": "some_name"
+        }
+      ]
+    }
+  ]
+}
+"""
+
+BAD_IMPORT_WRONG_KEY = """
+{
+  "imports": [
+    {
+      "name": "frameworks/base/services/core/java/com/android/server/am"
+    }
+  ]
+}
+"""
+
+BAD_IMPORT_WRONG_IMPORT_VALUE = """
+{
+  "imports": [
+    {
+      "path": "frameworks/base/services/core/java/com/android/server/am",
+      "option": "something"
+    }
+  ]
+}
+"""
+
+BAD_FILE_PATTERNS = """
+{
+  "presubmit": [
+    {
+      "name": "CtsWindowManagerDeviceTestCases",
+      "file_patterns": ["pattern", 123]
+    }
+  ]
+}
+"""
+
+TEST_MAPPING_WITH_SUPPORTED_COMMENTS = r"""
+// supported comment
+{
+  // supported comment!@#$%^&*()_
+  "presubmit": [
+    {
+      "name": "CtsWindowManagerDeviceTestCases\"foo//baz",
+      "options": [
+        // supported comment!@#$%^&*()_
+        {
+          "include-annotation": "android.platform.test.annotations.Presubmit"
+        }
+      ]
+    }
+  ],
+  "imports": [
+    {
+      "path": "path1//path2//path3"
+    }
+  ]
+}
+"""
+
+TEST_MAPPING_WITH_NON_SUPPORTED_COMMENTS = """
+{ #non-supported comments
+  // supported comments
+  "presubmit": [#non-supported comments
+    {  // non-supported comments
+      "name": "CtsWindowManagerDeviceTestCases",
+    }
+  ]
+}
+"""
+
+
+class AndroidTestMappingFormatTests(unittest.TestCase):
+    """Unittest for android_test_mapping_format module."""
+
+    def setUp(self):
+        self.tempdir = tempfile.mkdtemp()
+        self.test_mapping_file = os.path.join(self.tempdir, 'TEST_MAPPING')
+
+    def tearDown(self):
+        shutil.rmtree(self.tempdir)
+
+    def test_valid_test_mapping(self):
+        """Verify that the check doesn't raise any error for valid test mapping.
+        """
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(VALID_TEST_MAPPING)
+        with open(self.test_mapping_file, 'r') as f:
+            android_test_mapping_format.process_file(f.read())
+
+    def test_invalid_test_mapping_bad_json(self):
+        """Verify that TEST_MAPPING file with bad json can be detected."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(BAD_JSON)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                ValueError, android_test_mapping_format.process_file,
+                f.read())
+
+    def test_invalid_test_mapping_wrong_test_key(self):
+        """Verify that test config using wrong key can be detected."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(BAD_TEST_WRONG_KEY)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                android_test_mapping_format.InvalidTestMappingError,
+                android_test_mapping_format.process_file,
+                f.read())
+
+    def test_invalid_test_mapping_wrong_test_value(self):
+        """Verify that test config using wrong host value can be detected."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(BAD_TEST_WRONG_HOST_VALUE)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                android_test_mapping_format.InvalidTestMappingError,
+                android_test_mapping_format.process_file,
+                f.read())
+
+    def test_invalid_test_mapping_wrong_preferred_targets_value(self):
+        """Verify invalid preferred_targets are rejected."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(BAD_TEST_WRONG_PREFERRED_TARGETS_VALUE_NONE_LIST)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                android_test_mapping_format.InvalidTestMappingError,
+                android_test_mapping_format.process_file,
+                f.read())
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(BAD_TEST_WRONG_PREFERRED_TARGETS_VALUE_WRONG_TYPE)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                android_test_mapping_format.InvalidTestMappingError,
+                android_test_mapping_format.process_file,
+                f.read())
+
+    def test_invalid_test_mapping_wrong_test_option(self):
+        """Verify that test config using wrong option can be detected."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(BAD_TEST_WRONG_OPTION)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                android_test_mapping_format.InvalidTestMappingError,
+                android_test_mapping_format.process_file,
+                f.read())
+
+    def test_invalid_test_mapping_wrong_import_key(self):
+        """Verify that import setting using wrong key can be detected."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(BAD_IMPORT_WRONG_KEY)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                android_test_mapping_format.InvalidTestMappingError,
+                android_test_mapping_format.process_file,
+                f.read())
+
+    def test_invalid_test_mapping_wrong_import_value(self):
+        """Verify that import setting using wrong value can be detected."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(BAD_IMPORT_WRONG_IMPORT_VALUE)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                android_test_mapping_format.InvalidTestMappingError,
+                android_test_mapping_format.process_file,
+                f.read())
+
+    def test_invalid_test_mapping_file_patterns_value(self):
+        """Verify that file_patterns using wrong value can be detected."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(BAD_FILE_PATTERNS)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                android_test_mapping_format.InvalidTestMappingError,
+                android_test_mapping_format.process_file,
+                f.read())
+
+    def test_valid_test_mapping_file_with_supported_comments(self):
+        """Verify that '//'-format comment can be filtered."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(TEST_MAPPING_WITH_SUPPORTED_COMMENTS)
+        with open(self.test_mapping_file, 'r') as f:
+            android_test_mapping_format.process_file(f.read())
+
+    def test_valid_test_mapping_file_with_non_supported_comments(self):
+        """Verify that non-supported comment can be detected."""
+        with open(self.test_mapping_file, 'w') as f:
+            f.write(TEST_MAPPING_WITH_NON_SUPPORTED_COMMENTS)
+        with open(self.test_mapping_file, 'r') as f:
+            self.assertRaises(
+                ValueError, android_test_mapping_format.process_file,
+                f.read())
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/tools/checkpatch.pl b/tools/checkpatch.pl
index 0147c91..2d42eb9 100755
--- a/tools/checkpatch.pl
+++ b/tools/checkpatch.pl
@@ -1,4 +1,4 @@
-#!/usr/bin/perl -w
+#!/usr/bin/env perl
 # (c) 2001, Dave Jones. (the file handling bit)
 # (c) 2005, Joel Schopp <jschopp@austin.ibm.com> (the ugly bit)
 # (c) 2007,2008, Andy Whitcroft <apw@uk.ibm.com> (new conditions, test suite)
@@ -6,6 +6,7 @@
 # Licensed under the terms of the GNU GPL License version 2
 
 use strict;
+use warnings;
 use POSIX;
 use File::Basename;
 use Cwd 'abs_path';
@@ -27,12 +28,15 @@ my $emacs = 0;
 my $terse = 0;
 my $showfile = 0;
 my $file = 0;
+my $git = 0;
+my %git_commits = ();
 my $check = 0;
 my $check_orig = 0;
 my $summary = 1;
 my $mailback = 0;
 my $summary_file = 0;
 my $show_types = 0;
+my $list_types = 0;
 my $fix = 0;
 my $fix_inplace = 0;
 my $root;
@@ -51,7 +55,10 @@ my $min_conf_desc_length = 4;
 my $spelling_file = "$D/spelling.txt";
 my $codespell = 0;
 my $codespellfile = "/usr/share/codespell/dictionary.txt";
-my $color = 1;
+my $conststructsfile = "$D/const_structs.checkpatch";
+my $typedefsfile = "";
+my $color = "auto";
+my $allow_c99_comments = 1;
 
 sub help {
 	my ($exitcode) = @_;
@@ -68,13 +75,24 @@ Options:
   --emacs                    emacs compile window format
   --terse                    one line per report
   --showfile                 emit diffed file position, not input file position
+  -g, --git                  treat FILE as a single commit or git revision range
+                             single git commit with:
+                               <rev>
+                               <rev>^
+                               <rev>~n
+                             multiple git commits with:
+                               <rev1>..<rev2>
+                               <rev1>...<rev2>
+                               <rev>-<count>
+                             git merges are ignored
   -f, --file                 treat FILE as regular source file
   --subjective, --strict     enable more subjective tests
+  --list-types               list the possible message types
   --types TYPE(,TYPE2...)    show only these comma separated message types
   --ignore TYPE(,TYPE2...)   ignore various comma separated message types
+  --show-types               show the specific message type in the output
   --max-line-length=n        set the maximum line length, if exceeded, warn
   --min-conf-desc-length=n   set the min description length, if shorter, warn
-  --show-types               show the message "types" in the output
   --root=PATH                PATH to the kernel tree root
   --no-summary               suppress the per-file summary
   --mailback                 only produce a report in case of warnings/errors
@@ -97,7 +115,9 @@ Options:
   --codespell                Use the codespell dictionary for spelling/typos
                              (default:/usr/share/codespell/dictionary.txt)
   --codespellfile            Use this codespell dictionary
-  --color                    Use colors when output is STDOUT (default: on)
+  --typedefsfile             Read additional types from this file
+  --color[=WHEN]             Use colors 'always', 'never', or only when output
+                             is a terminal ('auto'). Default is 'auto'.
   -h, --help, --version      display this help and exit
 
 When FILE is - read standard input.
@@ -106,6 +126,38 @@ EOM
 	exit($exitcode);
 }
 
+sub uniq {
+	my %seen;
+	return grep { !$seen{$_}++ } @_;
+}
+
+sub list_types {
+	my ($exitcode) = @_;
+
+	my $count = 0;
+
+	local $/ = undef;
+
+	open(my $script, '<', abs_path($P)) or
+	    die "$P: Can't read '$P' $!\n";
+
+	my $text = <$script>;
+	close($script);
+
+	my @types = ();
+	# Also catch when type or level is passed through a variable
+	for ($text =~ /(?:(?:\bCHK|\bWARN|\bERROR|&\{\$msg_level})\s*\(|\$msg_type\s*=)\s*"([^"]+)"/g) {
+		push (@types, $_);
+	}
+	@types = sort(uniq(@types));
+	print("#\tMessage type\n\n");
+	foreach my $type (@types) {
+		print(++$count . "\t" . $type . "\n");
+	}
+
+	exit($exitcode);
+}
+
 my $conf = which_conf($configuration_file);
 if (-f $conf) {
 	my @conf_args;
@@ -132,6 +184,14 @@ if (-f $conf) {
 	unshift(@ARGV, @conf_args) if @conf_args;
 }
 
+# Perl's Getopt::Long allows options to take optional arguments after a space.
+# Prevent --color by itself from consuming other arguments
+foreach (@ARGV) {
+	if ($_ eq "--color" || $_ eq "-color") {
+		$_ = "--color=$color";
+	}
+}
+
 GetOptions(
 	'q|quiet+'	=> \$quiet,
 	'tree!'		=> \$tree,
@@ -141,11 +201,13 @@ GetOptions(
 	'terse!'	=> \$terse,
 	'showfile!'	=> \$showfile,
 	'f|file!'	=> \$file,
+	'g|git!'	=> \$git,
 	'subjective!'	=> \$check,
 	'strict!'	=> \$check,
 	'ignore=s'	=> \@ignore,
 	'types=s'	=> \@use,
 	'show-types!'	=> \$show_types,
+	'list-types!'	=> \$list_types,
 	'max-line-length=i' => \$max_line_length,
 	'min-conf-desc-length=i' => \$min_conf_desc_length,
 	'root=s'	=> \$root,
@@ -159,13 +221,18 @@ GetOptions(
 	'test-only=s'	=> \$tst_only,
 	'codespell!'	=> \$codespell,
 	'codespellfile=s'	=> \$codespellfile,
-	'color!'	=> \$color,
+	'typedefsfile=s'	=> \$typedefsfile,
+	'color=s'	=> \$color,
+	'no-color'	=> \$color,	#keep old behaviors of -nocolor
+	'nocolor'	=> \$color,	#keep old behaviors of -nocolor
 	'h|help'	=> \$help,
 	'version'	=> \$help
 ) or help(1);
 
 help(0) if ($help);
 
+list_types(0) if ($list_types);
+
 $fix = 1 if ($fix_inplace);
 $check_orig = $check;
 
@@ -178,9 +245,21 @@ if ($^V && $^V lt $minimum_perl_version) {
 	}
 }
 
+#if no filenames are given, push '-' to read patch from stdin
 if ($#ARGV < 0) {
-	print "$P: no input files\n";
-	exit(1);
+	push(@ARGV, '-');
+}
+
+if ($color =~ /^[01]$/) {
+	$color = !$color;
+} elsif ($color =~ /^always$/i) {
+	$color = 1;
+} elsif ($color =~ /^never$/i) {
+	$color = 0;
+} elsif ($color =~ /^auto$/i) {
+	$color = (-t STDOUT);
+} else {
+	die "Invalid color mode: $color\n";
 }
 
 sub hash_save_array_words {
@@ -264,12 +343,12 @@ our $Sparse	= qr{
 			__kernel|
 			__force|
 			__iomem|
-			__pmem|
 			__must_check|
 			__init_refok|
 			__kprobes|
 			__ref|
-			__rcu
+			__rcu|
+			__private
 		}x;
 our $InitAttributePrefix = qr{__(?:mem|cpu|dev|net_|)};
 our $InitAttributeData = qr{$InitAttributePrefix(?:initdata\b)};
@@ -284,7 +363,7 @@ our $Attribute	= qr{
 			__percpu|
 			__nocast|
 			__safe|
-			__bitwise__|
+			__bitwise|
 			__packed__|
 			__packed2__|
 			__naked|
@@ -373,8 +452,9 @@ our $typeTypedefs = qr{(?x:
 our $zero_initializer = qr{(?:(?:0[xX])?0+$Int_type?|NULL|false)\b};
 
 our $logFunctions = qr{(?x:
-	printk(?:_ratelimited|_once|)|
+	printk(?:_ratelimited|_once|_deferred_once|_deferred|)|
 	(?:[a-z0-9]+_){1,2}(?:printk|emerg|alert|crit|err|warning|warn|notice|info|debug|dbg|vdbg|devel|cont|WARN)(?:_ratelimited|_once|)|
+	TP_printk|
 	WARN(?:_RATELIMIT|_ONCE|)|
 	panic|
 	MODULE_[A-Z_]+|
@@ -473,7 +553,11 @@ our @mode_permission_funcs = (
 	["module_param_array_named", 5],
 	["debugfs_create_(?:file|u8|u16|u32|u64|x8|x16|x32|x64|size_t|atomic_t|bool|blob|regset32|u32_array)", 2],
 	["proc_create(?:_data|)", 2],
-	["(?:CLASS|DEVICE|SENSOR)_ATTR", 2],
+	["(?:CLASS|DEVICE|SENSOR|SENSOR_DEVICE|IIO_DEVICE)_ATTR", 2],
+	["IIO_DEV_ATTR_[A-Z_]+", 1],
+	["SENSOR_(?:DEVICE_|)ATTR_2", 2],
+	["SENSOR_TEMPLATE(?:_2|)", 3],
+	["__ATTR", 2],
 );
 
 #Create a search pattern for all these functions to speed up a loop below
@@ -482,6 +566,7 @@ foreach my $entry (@mode_permission_funcs) {
 	$mode_perms_search .= '|' if ($mode_perms_search ne "");
 	$mode_perms_search .= $entry->[0];
 }
+$mode_perms_search = "(?:${mode_perms_search})";
 
 our $mode_perms_world_writable = qr{
 	S_IWUGO		|
@@ -491,6 +576,63 @@ our $mode_perms_world_writable = qr{
 	0[0-7][0-7][2367]
 }x;
 
+our %mode_permission_string_types = (
+	"S_IRWXU" => 0700,
+	"S_IRUSR" => 0400,
+	"S_IWUSR" => 0200,
+	"S_IXUSR" => 0100,
+	"S_IRWXG" => 0070,
+	"S_IRGRP" => 0040,
+	"S_IWGRP" => 0020,
+	"S_IXGRP" => 0010,
+	"S_IRWXO" => 0007,
+	"S_IROTH" => 0004,
+	"S_IWOTH" => 0002,
+	"S_IXOTH" => 0001,
+	"S_IRWXUGO" => 0777,
+	"S_IRUGO" => 0444,
+	"S_IWUGO" => 0222,
+	"S_IXUGO" => 0111,
+);
+
+#Create a search pattern for all these strings to speed up a loop below
+our $mode_perms_string_search = "";
+foreach my $entry (keys %mode_permission_string_types) {
+	$mode_perms_string_search .= '|' if ($mode_perms_string_search ne "");
+	$mode_perms_string_search .= $entry;
+}
+our $single_mode_perms_string_search = "(?:${mode_perms_string_search})";
+our $multi_mode_perms_string_search = qr{
+	${single_mode_perms_string_search}
+	(?:\s*\|\s*${single_mode_perms_string_search})*
+}x;
+
+sub perms_to_octal {
+	my ($string) = @_;
+
+	return trim($string) if ($string =~ /^\s*0[0-7]{3,3}\s*$/);
+
+	my $val = "";
+	my $oval = "";
+	my $to = 0;
+	my $curpos = 0;
+	my $lastpos = 0;
+	while ($string =~ /\b(($single_mode_perms_string_search)\b(?:\s*\|\s*)?\s*)/g) {
+		$curpos = pos($string);
+		my $match = $2;
+		my $omatch = $1;
+		last if ($lastpos > 0 && ($curpos - length($omatch) != $lastpos));
+		$lastpos = $curpos;
+		$to |= $mode_permission_string_types{$match};
+		$val .= '\s*\|\s*' if ($val ne "");
+		$val .= $match;
+		$oval .= $omatch;
+	}
+	$oval =~ s/^\s*\|\s*//;
+	$oval =~ s/\s*\|\s*$//;
+	return sprintf("%04o", $to);
+}
+
 our $allowed_asm_includes = qr{(?x:
 	irq|
 	memory|
@@ -548,6 +690,44 @@ if ($codespell) {
 
 $misspellings = join("|", sort keys %spelling_fix) if keys %spelling_fix;
 
+sub read_words {
+	my ($wordsRef, $file) = @_;
+
+	if (open(my $words, '<', $file)) {
+		while (<$words>) {
+			my $line = $_;
+
+			$line =~ s/\s*\n?$//g;
+			$line =~ s/^\s*//g;
+
+			next if ($line =~ m/^\s*#/);
+			next if ($line =~ m/^\s*$/);
+			if ($line =~ /\s/) {
+				print("$file: '$line' invalid - ignored\n");
+				next;
+			}
+
+			$$wordsRef .= '|' if ($$wordsRef ne "");
+			$$wordsRef .= $line;
+		}
+		close($file);
+		return 1;
+	}
+
+	return 0;
+}
+
+my $const_structs = "";
+read_words(\$const_structs, $conststructsfile)
+    or warn "No structs that should be const will be found - file '$conststructsfile': $!\n";
+
+my $typeOtherTypedefs = "";
+if (length($typedefsfile)) {
+	read_words(\$typeOtherTypedefs, $typedefsfile)
+	    or warn "No additional types will be considered - file '$typedefsfile': $!\n";
+}
+$typeTypedefs .= '|' . $typeOtherTypedefs if ($typeOtherTypedefs ne "");
+
 sub build_types {
 	my $mods = "(?x:  \n" . join("|\n  ", (@modifierList, @modifierListFile)) . "\n)";
 	my $all = "(?x:  \n" . join("|\n  ", (@typeList, @typeListFile)) . "\n)";
@@ -610,8 +790,9 @@ our $FuncArg = qr{$Typecast{0,1}($LvalOrFunc|$Constant|$String)};
 
 our $declaration_macros = qr{(?x:
 	(?:$Storage\s+)?(?:[A-Z_][A-Z0-9]*_){0,2}(?:DEFINE|DECLARE)(?:_[A-Z0-9]+){1,6}\s*\(|
-	(?:$Storage\s+)?LIST_HEAD\s*\(|
-	(?:$Storage\s+)?${Type}\s+uninitialized_var\s*\(
+	(?:$Storage\s+)?[HLP]?LIST_HEAD\s*\(|
+	(?:$Storage\s+)?${Type}\s+uninitialized_var\s*\(|
+	(?:SKCIPHER_REQUEST|SHASH_DESC|AHASH_REQUEST)_ON_STACK\s*\(
 )};
 
 sub deparenthesize {
@@ -654,6 +835,16 @@ sub seed_camelcase_file {
 	}
 }
 
+sub is_maintained_obsolete {
+	my ($filename) = @_;
+
+	return 0 if (!$tree || !(-e "$root/scripts/get_maintainer.pl"));
+
+	my $status = `perl $root/scripts/get_maintainer.pl --status --nom --nol --nogit --nogit-fallback -f $filename 2>&1`;
+
+	return $status =~ /obsolete/i;
+}
+
 my $camelcase_seeded = 0;
 sub seed_camelcase_includes {
 	return if ($camelcase_seeded);
@@ -734,6 +925,7 @@ sub git_commit_info {
 #		    echo "commit $(cut -c 1-12,41-)"
 #		done
 	} elsif ($lines[0] =~ /^fatal: ambiguous argument '$commit': unknown revision or path not in the working tree\./) {
+		$id = undef;
 	} else {
 		$id = substr($lines[0], 0, 12);
 		$desc = substr($lines[0], 41);
@@ -751,10 +943,42 @@ my @fixed_inserted = ();
 my @fixed_deleted = ();
 my $fixlinenr = -1;
 
+# If input is git commits, extract all commits from the commit expressions.
+# For example, HEAD-3 means we need check 'HEAD, HEAD~1, HEAD~2'.
+die "$P: No git repository found\n" if ($git && !-e ".git");
+
+if ($git) {
+	my @commits = ();
+	foreach my $commit_expr (@ARGV) {
+		my $git_range;
+		if ($commit_expr =~ m/^(.*)-(\d+)$/) {
+			$git_range = "-$2 $1";
+		} elsif ($commit_expr =~ m/\.\./) {
+			$git_range = "$commit_expr";
+		} else {
+			$git_range = "-1 $commit_expr";
+		}
+		my $lines = `git log --no-color --no-merges --pretty=format:'%H %s' $git_range`;
+		foreach my $line (split(/\n/, $lines)) {
+			$line =~ /^([0-9a-fA-F]{40,40}) (.*)$/;
+			next if (!defined($1) || !defined($2));
+			my $sha1 = $1;
+			my $subject = $2;
+			unshift(@commits, $sha1);
+			$git_commits{$sha1} = $subject;
+		}
+	}
+	die "$P: no git commits after extraction!\n" if (@commits == 0);
+	@ARGV = @commits;
+}
+
 my $vname;
 for my $filename (@ARGV) {
 	my $FILE;
-	if ($file) {
+	if ($git) {
+		open($FILE, '-|', "git format-patch -M --stdout -1 $filename") ||
+			die "$P: $filename: git format-patch failed - $!\n";
+	} elsif ($file) {
 		open($FILE, '-|', "diff -u /dev/null $filename") ||
 			die "$P: $filename: diff failed - $!\n";
 	} elsif ($filename eq '-') {
@@ -765,6 +989,8 @@ for my $filename (@ARGV) {
 	}
 	if ($filename eq '-') {
 		$vname = 'Your patch';
+	} elsif ($git) {
+		$vname = "Commit " . substr($filename, 0, 12) . ' ("' . $git_commits{$filename} . '")';
 	} else {
 		$vname = $filename;
 	}
@@ -850,7 +1076,7 @@ sub parse_email {
 	} elsif ($formatted_email =~ /(\S+\@\S+)(.*)$/) {
 		$address = $1;
 		$comment = $2 if defined $2;
-		$formatted_email =~ s/$address.*$//;
+		$formatted_email =~ s/\Q$address\E.*$//;
 		$name = $formatted_email;
 		$name = trim($name);
 		$name =~ s/^\"|\"$//g;
@@ -992,7 +1218,7 @@ sub sanitise_line {
 	for ($off = 1; $off < length($line); $off++) {
 		$c = substr($line, $off, 1);
 
-		# Comments we are wacking completly including the begin
+		# Comments we are whacking completely including the begin
 		# and end, all to $;.
 		if ($sanitise_quote eq '' && substr($line, $off, 2) eq '/*') {
 			$sanitise_quote = '*/';
@@ -1061,12 +1287,18 @@ sub sanitise_line {
 		$res =~ s@(\#\s*(?:error|warning)\s+).*@$1$clean@;
 	}
 
+	if ($allow_c99_comments && $res =~ m@(//.*$)@) {
+		my $match = $1;
+		$res =~ s/\Q$match\E/"$;" x length($match)/e;
+	}
+
 	return $res;
 }
 
 sub get_quoted_string {
 	my ($line, $rawline) = @_;
 
+	return "" if (!defined($line) || !defined($rawline));
 	return "" if ($line !~ m/($String)/g);
 	return substr($rawline, $-[0], $+[0] - $-[0]);
 }
@@ -1414,6 +1646,28 @@ sub raw_line {
 	return $line;
 }
 
+sub get_stat_real {
+	my ($linenr, $lc) = @_;
+
+	my $stat_real = raw_line($linenr, 0);
+	for (my $count = $linenr + 1; $count <= $lc; $count++) {
+		$stat_real = $stat_real . "\n" . raw_line($count, 0);
+	}
+
+	return $stat_real;
+}
+
+sub get_stat_here {
+	my ($linenr, $cnt, $here) = @_;
+
+	my $herectx = $here . "\n";
+	for (my $n = 0; $n < $cnt; $n++) {
+		$herectx .= raw_line($linenr, $n) . "\n";
+	}
+
+	return $herectx;
+}
+
 sub cat_vet {
 	my ($vet) = @_;
 	my ($res, $coded);
@@ -1695,6 +1949,8 @@ my $prefix = '';
 sub show_type {
 	my ($type) = @_;
 
+	$type =~ tr/[a-z]/[A-Z]/;
+
 	return defined $use_type{$type} if (scalar keys %use_type > 0);
 
 	return !defined $ignore_type{$type};
@@ -1708,7 +1964,7 @@ sub report {
 		return 0;
 	}
 	my $output = '';
-	if (-t STDOUT && $color) {
+	if ($color) {
 		if ($level eq 'ERROR') {
 			$output .= RED;
 		} elsif ($level eq 'WARNING') {
@@ -1719,10 +1975,10 @@ sub report {
 	}
 	$output .= $prefix . $level . ':';
 	if ($show_types) {
-		$output .= BLUE if (-t STDOUT && $color);
+		$output .= BLUE if ($color);
 		$output .= "$type:";
 	}
-	$output .= RESET if (-t STDOUT && $color);
+	$output .= RESET if ($color);
 	$output .= ' ' . $msg . "\n";
 
 	if ($showfile) {
@@ -1980,7 +2236,8 @@ sub process {
 	my $is_patch = 0;
 	my $in_header_lines = $file ? 0 : 1;
 	my $in_commit_log = 0;		#Scanning lines before patch
-       my $commit_log_possible_stack_dump = 0;
+	my $has_commit_log = 0;		#Encountered lines before patch
+	my $commit_log_possible_stack_dump = 0;
 	my $commit_log_long_line = 0;
 	my $commit_log_has_diff = 0;
 	my $reported_maintainer_file = 0;
@@ -2000,6 +2257,7 @@ sub process {
 	my $realline = 0;
 	my $realcnt = 0;
 	my $here = '';
+	my $context_function;		#undef'd unless there's a known function
 	my $in_comment = 0;
 	my $comment_edge = 0;
 	my $first_line = 0;
@@ -2023,6 +2281,8 @@ sub process {
 
 	my $camelcase_file_seeded = 0;
 
+	my $checklicenseline = 1;
+
 	sanitise_line_reset();
 	my $line;
 	foreach my $rawline (@rawlines) {
@@ -2033,12 +2293,12 @@ sub process {
 
 		if ($rawline=~/^\+\+\+\s+(\S+)/) {
 			$setup_docs = 0;
-			if ($1 =~ m@Documentation/kernel-parameters.txt$@) {
+			if ($1 =~ m@Documentation/admin-guide/kernel-parameters.rst$@) {
 				$setup_docs = 1;
 			}
 			#next;
 		}
-		if ($rawline=~/^\@\@ -\d+(?:,\d+)? \+(\d+)(,(\d+))? \@\@/) {
+		if ($rawline =~ /^\@\@ -\d+(?:,\d+)? \+(\d+)(,(\d+))? \@\@/) {
 			$realline=$1-1;
 			if (defined $2) {
 				$realcnt=$3+1;
@@ -2117,7 +2377,8 @@ sub process {
 
 #extract the line range in the file after the patch is applied
 		if (!$in_commit_log &&
-		    $line =~ /^\@\@ -\d+(?:,\d+)? \+(\d+)(,(\d+))? \@\@/) {
+		    $line =~ /^\@\@ -\d+(?:,\d+)? \+(\d+)(,(\d+))? \@\@(.*)/) {
+			my $context = $4;
 			$is_patch = 1;
 			$first_line = $linenr + 1;
 			$realline=$1-1;
@@ -2133,6 +2394,11 @@ sub process {
 			%suppress_whiletrailers = ();
 			%suppress_export = ();
 			$suppress_statement = 0;
+			if ($context =~ /\b(\w+)\s*\(/) {
+				$context_function = $1;
+			} else {
+				undef $context_function;
+			}
 			next;
 
 # track the line number as we move through the hunk, note that
@@ -2199,11 +2465,16 @@ sub process {
 		}
 
 		if ($found_file) {
+			if (is_maintained_obsolete($realfile)) {
+				WARN("OBSOLETE",
+				     "$realfile is marked as 'obsolete' in the MAINTAINERS hierarchy.  No unnecessary modifications please.\n");
+			}
 			if ($realfile =~ m@^(?:drivers/net/|net/|drivers/staging/)@) {
 				$check = 1;
 			} else {
 				$check = $check_orig;
 			}
+			$checklicenseline = 1;
 			next;
 		}
 
@@ -2370,8 +2641,10 @@ sub process {
 
 # Check for git id commit length and improperly formed commit descriptions
 		if ($in_commit_log && !$commit_log_possible_stack_dump &&
+		    $line !~ /^\s*(?:Link|Patchwork|http|https|BugLink):/i &&
+		    $line !~ /^This reverts commit [0-9a-f]{7,40}/ &&
 		    ($line =~ /\bcommit\s+[0-9a-f]{5,}\b/i ||
-		     ($line =~ /\b[0-9a-f]{12,40}\b/i &&
+		     ($line =~ /(?:\s|^)[0-9a-f]{12,40}(?:[\s"'\(\[]|$)/i &&
 		      $line !~ /[\<\[][0-9a-f]{12,40}[\>\]]/i &&
 		      $line !~ /\bfixes:\s*[0-9a-f]{12,40}/i))) {
 			my $init_char = "c";
@@ -2418,7 +2691,8 @@ sub process {
 			($id, $description) = git_commit_info($orig_commit,
 							      $id, $orig_desc);
 
-			if ($short || $long || $space || $case || ($orig_desc ne $description) || !$hasparens) {
+			if (defined($id) &&
+			   ($short || $long || $space || $case || ($orig_desc ne $description) || !$hasparens)) {
 				ERROR("GIT_COMMIT_ID",
 				      "Please use git commit description style 'commit <12+ chars of sha1> (\"<title line>\")' - ie: '${init_char}ommit $id (\"$description\")'\n" . $herecurr);
 			}
@@ -2430,6 +2704,7 @@ sub process {
 		     $line =~ /^rename (?:from|to) [\w\/\.\-]+\s*$/ ||
 		     ($line =~ /\{\s*([\w\/\.\-]*)\s*\=\>\s*([\w\/\.\-]*)\s*\}/ &&
 		      (defined($1) || defined($2))))) {
+			$is_patch = 1;
 			$reported_maintainer_file = 1;
 			WARN("FILE_PATH_CHANGES",
 			     "added, moved or deleted file(s), does MAINTAINERS need updating?\n" . $herecurr);
@@ -2442,20 +2717,6 @@ sub process {
 				$herecurr) if (!$emitted_corrupt++);
 		}
 
-# Check for absolute kernel paths.
-		if ($tree) {
-			while ($line =~ m{(?:^|\s)(/\S*)}g) {
-				my $file = $1;
-
-				if ($file =~ m{^(.*?)(?::\d+)+:?$} &&
-				    check_absolute_file($1, $herecurr)) {
-					#
-				} else {
-					check_absolute_file($file, $herecurr);
-				}
-			}
-		}
-
 # UTF-8 regex found at http://www.w3.org/International/questions/qa-forms-utf-8.en.php
 		if (($realfile =~ /^$/ || $line =~ /^\+/) &&
 		    $rawline !~ m/^$UTF8*$/) {
@@ -2472,10 +2733,11 @@ sub process {
 # Check if it's the start of a commit log
 # (not a header line and we haven't seen the patch filename)
 		if ($in_header_lines && $realfile =~ /^$/ &&
-		    !($rawline =~ /^\s+\S/ ||
-		      $rawline =~ /^(commit\b|from\b|[\w-]+:).*$/i)) {
+		    !($rawline =~ /^\s+(?:\S|$)/ ||
+		      $rawline =~ /^(?:commit\b|from\b|[\w-]+:)/i)) {
 			$in_header_lines = 0;
 			$in_commit_log = 1;
+			$has_commit_log = 1;
 		}
 
 # Check if there is UTF-8 in a commit log when a mail header has explicitly
@@ -2492,6 +2754,20 @@ sub process {
 			    "8-bit UTF-8 used in possible commit log\n" . $herecurr);
 		}
 
+# Check for absolute kernel paths in commit message
+		if ($tree && $in_commit_log) {
+			while ($line =~ m{(?:^|\s)(/\S*)}g) {
+				my $file = $1;
+
+				if ($file =~ m{^(.*?)(?::\d+)+:?$} &&
+				    check_absolute_file($1, $herecurr)) {
+					#
+				} else {
+					check_absolute_file($file, $herecurr);
+				}
+			}
+		}
+
 # Check for various typo / spelling mistakes
 		if (defined($misspellings) &&
 		    ($in_commit_log || $line =~ /^(?:\+|Subject:)/i)) {
@@ -2500,10 +2776,10 @@ sub process {
 				my $typo_fix = $spelling_fix{lc($typo)};
 				$typo_fix = ucfirst($typo_fix) if ($typo =~ /^[A-Z]/);
 				$typo_fix = uc($typo_fix) if ($typo =~ /^[A-Z]+$/);
-				my $msg_type = \&WARN;
-				$msg_type = \&CHK if ($file);
-				if (&{$msg_type}("TYPO_SPELLING",
-						 "'$typo' may be misspelled - perhaps '$typo_fix'?\n" . $herecurr) &&
+				my $msg_level = \&WARN;
+				$msg_level = \&CHK if ($file);
+				if (&{$msg_level}("TYPO_SPELLING",
+						  "'$typo' may be misspelled - perhaps '$typo_fix'?\n" . $herecurr) &&
 				    $fix) {
 					$fixed[$fixlinenr] =~ s/(^|[^A-Za-z@])($typo)($|[^A-Za-z@])/$1$typo_fix$3/;
 				}
@@ -2534,20 +2810,24 @@ sub process {
 
 # Check for FSF mailing addresses.
 		if ($rawline =~ /\bwrite to the Free/i ||
+		    $rawline =~ /\b675\s+Mass\s+Ave/i ||
 		    $rawline =~ /\b59\s+Temple\s+Pl/i ||
 		    $rawline =~ /\b51\s+Franklin\s+St/i) {
 			my $herevet = "$here\n" . cat_vet($rawline) . "\n";
-			my $msg_type = \&ERROR;
-			$msg_type = \&CHK if ($file);
-			&{$msg_type}("FSF_MAILING_ADDRESS",
-				     "Do not include the paragraph about writing to the Free Software Foundation's mailing address from the sample GPL notice. The FSF has changed addresses in the past, and may do so again. Linux already includes a copy of the GPL.\n" . $herevet)
+			my $msg_level = \&ERROR;
+			$msg_level = \&CHK if ($file);
+			&{$msg_level}("FSF_MAILING_ADDRESS",
+				      "Do not include the paragraph about writing to the Free Software Foundation's mailing address from the sample GPL notice. The FSF has changed addresses in the past, and may do so again. Linux already includes a copy of the GPL.\n" . $herevet)
 		}
 
 # check for Kconfig help text having a real description
 # Only applies when adding the entry originally, after that we do not have
 # sufficient context to determine whether it is indeed long enough.
 		if ($realfile =~ /Kconfig/ &&
-		    $line =~ /^\+\s*config\s+/) {
+		    # 'choice' is usually the last thing on the line (though
+		    # Kconfig supports named choices), so use a word boundary
+		    # (\b) rather than a whitespace character (\s)
+		    $line =~ /^\+\s*(?:config|menuconfig|choice)\b/) {
 			my $length = 0;
 			my $cnt = $realcnt;
 			my $ln = $linenr + 1;
@@ -2562,9 +2842,13 @@ sub process {
 				next if ($f =~ /^-/);
 				last if (!$file && $f =~ /^\@\@/);
 
-				if ($lines[$ln - 1] =~ /^\+\s*(?:bool|tristate)\s*\"/) {
+				if ($lines[$ln - 1] =~ /^\+\s*(?:bool|tristate|prompt)\s*["']/) {
 					$is_start = 1;
-				} elsif ($lines[$ln - 1] =~ /^\+\s*(?:---)?help(?:---)?$/) {
+				} elsif ($lines[$ln - 1] =~ /^\+\s*(?:help|---help---)\s*$/) {
+					if ($lines[$ln - 1] =~ "---help---") {
+						WARN("CONFIG_DESCRIPTION",
+						     "prefer 'help' over '---help---' for new help texts\n" . $herecurr);
+					}
 					$length = -1;
 				}
 
@@ -2572,7 +2856,13 @@ sub process {
 				$f =~ s/#.*//;
 				$f =~ s/^\s+//;
 				next if ($f =~ /^$/);
-				if ($f =~ /^\s*config\s/) {
+
+				# This only checks context lines in the patch
+				# and so hopefully shouldn't trigger false
+				# positives, even though some of these are
+				# common words in help texts
+				if ($f =~ /^\s*(?:config|menuconfig|choice|endchoice|
+						  if|endif|menu|endmenu|source)\b/x) {
 					$is_end = 1;
 					last;
 				}
@@ -2585,11 +2875,15 @@ sub process {
 			#print "is_start<$is_start> is_end<$is_end> length<$length>\n";
 		}
 
-# discourage the addition of CONFIG_EXPERIMENTAL in Kconfig.
-		if ($realfile =~ /Kconfig/ &&
-		    $line =~ /.\s*depends on\s+.*\bEXPERIMENTAL\b/) {
-			WARN("CONFIG_EXPERIMENTAL",
-			     "Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see https://lkml.org/lkml/2012/10/23/580\n");
+# check for MAINTAINERS entries that don't have the right form
+		if ($realfile =~ /^MAINTAINERS$/ &&
+		    $rawline =~ /^\+[A-Z]:/ &&
+		    $rawline !~ /^\+[A-Z]:\t\S/) {
+			if (WARN("MAINTAINERS_STYLE",
+				 "MAINTAINERS entries use one tab after TYPE:\n" . $herecurr) &&
+			    $fix) {
+				$fixed[$fixlinenr] =~ s/^(\+[A-Z]):\s*/$1:\t/;
+			}
 		}
 
 # discourage the use of boolean for type definition attributes of Kconfig options
@@ -2644,8 +2938,32 @@ sub process {
 			}
 		}
 
+# check for using SPDX license tag at beginning of files
+		if ($realline == $checklicenseline) {
+			if ($rawline =~ /^[ \+]\s*\#\!\s*\//) {
+				$checklicenseline = 2;
+			} elsif ($rawline =~ /^\+/) {
+				my $comment = "";
+				if ($realfile =~ /\.(h|s|S)$/) {
+					$comment = '/*';
+				} elsif ($realfile =~ /\.(c|dts|dtsi)$/) {
+					$comment = '//';
+				} elsif (($checklicenseline == 2) || $realfile =~ /\.(sh|pl|py|awk|tc)$/) {
+					$comment = '#';
+				} elsif ($realfile =~ /\.rst$/) {
+					$comment = '..';
+				}
+
+				if ($comment !~ /^$/ &&
+				    $rawline !~ /^\+\Q$comment\E SPDX-License-Identifier: /) {
+					WARN("SPDX_LICENSE_TAG",
+					     "Missing or malformed SPDX-License-Identifier tag in line $checklicenseline\n" . $herecurr);
+				}
+			}
+		}
+
 # check we are in a valid source file if not then ignore this hunk
-		next if ($realfile !~ /\.(h|c|s|S|pl|sh|dtsi|dts)$/);
+		next if ($realfile !~ /\.(h|c|s|S|sh|dtsi|dts)$/);
 
 # line length limit (with some exclusions)
 #
@@ -2653,9 +2971,10 @@ sub process {
 #	logging functions like pr_info that end in a string
 #	lines with a single string
 #	#defines that are a single string
+#	lines with an RFC3986 like URL
 #
 # There are 3 different line length message types:
-# LONG_LINE_COMMENT	a comment starts before but extends beyond $max_linelength
+# LONG_LINE_COMMENT	a comment starts before but extends beyond $max_line_length
 # LONG_LINE_STRING	a string starts before but extends beyond $max_line_length
 # LONG_LINE		all other lines longer than $max_line_length
 #
@@ -2679,6 +2998,15 @@ sub process {
 				 $line =~ /^\+\s*#\s*define\s+\w+\s+$String$/) {
 				$msg_type = "";
 
+			# More special cases
+			} elsif ($line =~ /^\+.*\bEFI_GUID\s*\(/ ||
+				 $line =~ /^\+\s*(?:\w+)?\s*DEFINE_PER_CPU/) {
+				$msg_type = "";
+
+			# URL ($rawline is used in case the URL is in a comment)
+			} elsif ($rawline =~ /^\+.*\b[a-z][\w\.\+\-]*:\/\/\S+/i) {
+				$msg_type = "";
+
 			# Otherwise set the alternate message types
 
 			# a comment starts before $max_line_length
@@ -2705,20 +3033,6 @@ sub process {
 			     "adding a line without newline at end of file\n" . $herecurr);
 		}
 
-# Blackfin: use hi/lo macros
-		if ($realfile =~ m@arch/blackfin/.*\.S$@) {
-			if ($line =~ /\.[lL][[:space:]]*=.*&[[:space:]]*0x[fF][fF][fF][fF]/) {
-				my $herevet = "$here\n" . cat_vet($line) . "\n";
-				ERROR("LO_MACRO",
-				      "use the LO() macro, not (... & 0xFFFF)\n" . $herevet);
-			}
-			if ($line =~ /\.[hH][[:space:]]*=.*>>[[:space:]]*16/) {
-				my $herevet = "$here\n" . cat_vet($line) . "\n";
-				ERROR("HI_MACRO",
-				      "use the HI() macro, not (... >> 16)\n" . $herevet);
-			}
-		}
-
 # check we are in a valid source file C or perl if not then ignore this hunk
 		next if ($realfile !~ /\.(h|c|pl|dtsi|dts)$/);
 
@@ -2748,15 +3062,34 @@ sub process {
 			}
 		}
 
+# check for assignments on the start of a line
+		if ($sline =~ /^\+\s+($Assignment)[^=]/) {
+			CHK("ASSIGNMENT_CONTINUATIONS",
+			    "Assignment operator '$1' should be on the previous line\n" . $hereprev);
+		}
+
 # check for && or || at the start of a line
 		if ($rawline =~ /^\+\s*(&&|\|\|)/) {
 			CHK("LOGICAL_CONTINUATIONS",
 			    "Logical continuations should be on the previous line\n" . $hereprev);
 		}
 
+# check indentation starts on a tab stop
+		if ($^V && $^V ge 5.10.0 &&
+		    $sline =~ /^\+\t+( +)(?:$c90_Keywords\b|\{\s*$|\}\s*(?:else\b|while\b|\s*$)|$Declare\s*$Ident\s*[;=])/) {
+			my $indent = length($1);
+			if ($indent % 8) {
+				if (WARN("TABSTOP",
+					 "Statements should start on a tabstop\n" . $herecurr) &&
+				    $fix) {
+					$fixed[$fixlinenr] =~ s@(^\+\t+) +@$1 . "\t" x ($indent/8)@e;
+				}
+			}
+		}
+
 # check multi-line statement indentation matches previous line
 		if ($^V && $^V ge 5.10.0 &&
-		    $prevline =~ /^\+([ \t]*)((?:$c90_Keywords(?:\s+if)\s*)|(?:$Declare\s*)?(?:$Ident|\(\s*\*\s*$Ident\s*\))\s*|$Ident\s*=\s*$Ident\s*)\(.*(\&\&|\|\||,)\s*$/) {
+		    $prevline =~ /^\+([ \t]*)((?:$c90_Keywords(?:\s+if)\s*)|(?:$Declare\s*)?(?:$Ident|\(\s*\*\s*$Ident\s*\))\s*|(?:\*\s*)*$Lval\s*=\s*$Ident\s*)\(.*(\&\&|\|\||,)\s*$/) {
 			$prevline =~ /^\+(\t*)(.*)$/;
 			my $oldindent = $1;
 			my $rest = $2;
@@ -2830,6 +3163,30 @@ sub process {
 			     "Block comments use a trailing */ on a separate line\n" . $herecurr);
 		}
 
+# Block comment * alignment
+		if ($prevline =~ /$;[ \t]*$/ &&			#ends in comment
+		    $line =~ /^\+[ \t]*$;/ &&			#leading comment
+		    $rawline =~ /^\+[ \t]*\*/ &&		#leading *
+		    (($prevrawline =~ /^\+.*?\/\*/ &&		#leading /*
+		      $prevrawline !~ /\*\/[ \t]*$/) ||		#no trailing */
+		     $prevrawline =~ /^\+[ \t]*\*/)) {		#leading *
+			my $oldindent;
+			$prevrawline =~ m@^\+([ \t]*/?)\*@;
+			if (defined($1)) {
+				$oldindent = expand_tabs($1);
+			} else {
+				$prevrawline =~ m@^\+(.*/?)\*@;
+				$oldindent = expand_tabs($1);
+			}
+			$rawline =~ m@^\+([ \t]*)\*@;
+			my $newindent = $1;
+			$newindent = expand_tabs($newindent);
+			if (length($oldindent) ne length($newindent)) {
+				WARN("BLOCK_COMMENT_STYLE",
+				     "Block comments should align the * on each line\n" . $hereprev);
+			}
+		}
+
 # check for missing blank lines after struct/union declarations
 # with exceptions for various attributes and macros
 		if ($prevline =~ /^[\+ ]};?\s*$/ &&
@@ -2841,6 +3198,7 @@ sub process {
 		      $line =~ /^\+[a-z_]*init/ ||
 		      $line =~ /^\+\s*(?:static\s+)?[A-Z_]*ATTR/ ||
 		      $line =~ /^\+\s*DECLARE/ ||
+		      $line =~ /^\+\s*builtin_[\w_]*driver/ ||
 		      $line =~ /^\+\s*__setup/)) {
 			if (CHK("LINE_SPACING",
 				"Please use a blank line after function/struct/union/enum declarations\n" . $hereprev) &&
@@ -2920,6 +3278,23 @@ sub process {
 # check we are in a valid C source file if not then ignore this hunk
 		next if ($realfile !~ /\.(h|c)$/);
 
+# check for unusual line ending [ or (
+		if ($line =~ /^\+.*([\[\(])\s*$/) {
+			CHK("OPEN_ENDED_LINE",
+			    "Lines should not end with a '$1'\n" . $herecurr);
+		}
+
+# check if this appears to be the start function declaration, save the name
+		if ($sline =~ /^\+\{\s*$/ &&
+		    $prevline =~ /^\+(?:(?:(?:$Storage|$Inline)\s*)*\s*$Type\s*)?($Ident)\(/) {
+			$context_function = $1;
+		}
+
+# check if this appears to be the end of function declaration
+		if ($sline =~ /^\+\}\s*$/) {
+			undef $context_function;
+		}
+
 # check indentation of any line with a bare else
 # (but not if it is a multiple line "if (foo) return bar; else return baz;")
 # if the previous line is a break or return and is indented 1 tab more...
@@ -2944,30 +3319,12 @@ sub process {
 			}
 		}
 
-# discourage the addition of CONFIG_EXPERIMENTAL in #if(def).
-		if ($line =~ /^\+\s*\#\s*if.*\bCONFIG_EXPERIMENTAL\b/) {
-			WARN("CONFIG_EXPERIMENTAL",
-			     "Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see https://lkml.org/lkml/2012/10/23/580\n");
-		}
-
 # check for RCS/CVS revision markers
 		if ($rawline =~ /^\+.*\$(Revision|Log|Id)(?:\$|)/) {
 			WARN("CVS_KEYWORD",
 			     "CVS style keyword markers, these will _not_ be updated\n". $herecurr);
 		}
 
-# Blackfin: don't use __builtin_bfin_[cs]sync
-		if ($line =~ /__builtin_bfin_csync/) {
-			my $herevet = "$here\n" . cat_vet($line) . "\n";
-			ERROR("CSYNC",
-			      "use the CSYNC() macro in asm/blackfin.h\n" . $herevet);
-		}
-		if ($line =~ /__builtin_bfin_ssync/) {
-			my $herevet = "$here\n" . cat_vet($line) . "\n";
-			ERROR("SSYNC",
-			      "use the SSYNC() macro in asm/blackfin.h\n" . $herevet);
-		}
-
 # check for old HOTPLUG __dev<foo> section markings
 		if ($line =~ /\b(__dev(init|exit)(data|const|))\b/) {
 			WARN("HOTPLUG_SECTION",
@@ -2978,7 +3335,7 @@ sub process {
 		my ($stat, $cond, $line_nr_next, $remain_next, $off_next,
 		    $realline_next);
 #print "LINE<$line>\n";
-		if ($linenr >= $suppress_statement &&
+		if ($linenr > $suppress_statement &&
 		    $realcnt && $sline =~ /.\s*\S/) {
 			($stat, $cond, $line_nr_next, $remain_next, $off_next) =
 				ctx_statement_block($linenr, $realcnt, 0);
@@ -3125,7 +3482,7 @@ sub process {
 		}
 
 # Check relative indent for conditionals and blocks.
-		if ($line =~ /\b(?:(?:if|while|for|(?:[a-z_]+|)for_each[a-z_]+)\s*\(|do\b)/ && $line !~ /^.\s*#/ && $line !~ /\}\s*while\s*/) {
+		if ($line =~ /\b(?:(?:if|while|for|(?:[a-z_]+|)for_each[a-z_]+)\s*\(|(?:do|else)\b)/ && $line !~ /^.\s*#/ && $line !~ /\}\s*while\s*/) {
 			($stat, $cond, $line_nr_next, $remain_next, $off_next) =
 				ctx_statement_block($linenr, $realcnt, 0)
 					if (!defined $stat);
@@ -3217,6 +3574,8 @@ sub process {
 			if ($check && $s ne '' &&
 			    (($sindent % 8) != 0 ||
 			     ($sindent < $indent) ||
+			     ($sindent == $indent &&
+			      ($s !~ /^\s*(?:\}|\{|else\b)/)) ||
 			     ($sindent > $indent + 8))) {
 				WARN("SUSPECT_CODE_INDENT",
 				     "suspect code indent for conditional statements ($indent, $sindent)\n" . $herecurr . "$stat_real\n");
@@ -3239,6 +3598,42 @@ sub process {
 #ignore lines not being added
 		next if ($line =~ /^[^\+]/);
 
+# check for dereferences that span multiple lines
+		if ($prevline =~ /^\+.*$Lval\s*(?:\.|->)\s*$/ &&
+		    $line =~ /^\+\s*(?!\#\s*(?!define\s+|if))\s*$Lval/) {
+			$prevline =~ /($Lval\s*(?:\.|->))\s*$/;
+			my $ref = $1;
+			$line =~ /^.\s*($Lval)/;
+			$ref .= $1;
+			$ref =~ s/\s//g;
+			WARN("MULTILINE_DEREFERENCE",
+			     "Avoid multiple line dereference - prefer '$ref'\n" . $hereprev);
+		}
+
+# check for declarations of signed or unsigned without int
+		while ($line =~ m{\b($Declare)\s*(?!char\b|short\b|int\b|long\b)\s*($Ident)?\s*[=,;\[\)\(]}g) {
+			my $type = $1;
+			my $var = $2;
+			$var = "" if (!defined $var);
+			if ($type =~ /^(?:(?:$Storage|$Inline|$Attribute)\s+)*((?:un)?signed)((?:\s*\*)*)\s*$/) {
+				my $sign = $1;
+				my $pointer = $2;
+
+				$pointer = "" if (!defined $pointer);
+
+				if (WARN("UNSPECIFIED_INT",
+					 "Prefer '" . trim($sign) . " int" . rtrim($pointer) . "' to bare use of '$sign" . rtrim($pointer) . "'\n" . $herecurr) &&
+				    $fix) {
+					my $decl = trim($sign) . " int ";
+					my $comp_pointer = $pointer;
+					$comp_pointer =~ s/\s//g;
+					$decl .= $comp_pointer;
+					$decl = rtrim($decl) if ($var eq "");
+					$fixed[$fixlinenr] =~ s@\b$sign\s*\Q$pointer\E\s*$var\b@$decl$var@;
+				}
+			}
+		}
+
 # TEST: allow direct testing of the type matcher.
 		if ($dbg_type) {
 			if ($line =~ /^.\s*$Declare\s*$/) {
@@ -3274,7 +3669,7 @@ sub process {
 				$fixedline =~ s/\s*=\s*$/ = {/;
 				fix_insert_line($fixlinenr, $fixedline);
 				$fixedline = $line;
-				$fixedline =~ s/^(.\s*){\s*/$1/;
+				$fixedline =~ s/^(.\s*)\{\s*/$1/;
 				fix_insert_line($fixlinenr, $fixedline);
 			}
 		}
@@ -3437,22 +3832,13 @@ sub process {
 			}
 		}
 
-# check for uses of DEFINE_PCI_DEVICE_TABLE
-		if ($line =~ /\bDEFINE_PCI_DEVICE_TABLE\s*\(\s*(\w+)\s*\)\s*=/) {
-			if (WARN("DEFINE_PCI_DEVICE_TABLE",
-				 "Prefer struct pci_device_id over deprecated DEFINE_PCI_DEVICE_TABLE\n" . $herecurr) &&
-			    $fix) {
-				$fixed[$fixlinenr] =~ s/\b(?:static\s+|)DEFINE_PCI_DEVICE_TABLE\s*\(\s*(\w+)\s*\)\s*=\s*/static const struct pci_device_id $1\[\] = /;
-			}
-		}
-
 # check for new typedefs, only function parameters and sparse annotations
 # make sense.
 		if ($line =~ /\btypedef\s/ &&
 		    $line !~ /\btypedef\s+$Type\s*\(\s*\*?$Ident\s*\)\s*\(/ &&
 		    $line !~ /\btypedef\s+$Type\s+$Ident\s*\(/ &&
 		    $line !~ /\b$typeTypedefs\b/ &&
-		    $line !~ /\b__bitwise(?:__|)\b/) {
+		    $line !~ /\b__bitwise\b/) {
 			WARN("NEW_TYPEDEFS",
 			     "do not add new typedefs\n" . $herecurr);
 		}
@@ -3515,10 +3901,10 @@ sub process {
 
 # avoid BUG() or BUG_ON()
 		if ($line =~ /\b(?:BUG|BUG_ON)\b/) {
-			my $msg_type = \&WARN;
-			$msg_type = \&CHK if ($file);
-			&{$msg_type}("AVOID_BUG",
-				     "Avoid crashing the kernel - try using WARN_ON & recovery code rather than BUG() or BUG_ON()\n" . $herecurr);
+			my $msg_level = \&WARN;
+			$msg_level = \&CHK if ($file);
+			&{$msg_level}("AVOID_BUG",
+				      "Avoid crashing the kernel - try using WARN_ON & recovery code rather than BUG() or BUG_ON()\n" . $herecurr);
 		}
 
 # avoid LINUX_VERSION_CODE
@@ -3533,28 +3919,10 @@ sub process {
 			     "Prefer printk_ratelimited or pr_<level>_ratelimited to printk_ratelimit\n" . $herecurr);
 		}
 
-# printk should use KERN_* levels.  Note that follow on printk's on the
-# same line do not need a level, so we use the current block context
-# to try and find and validate the current printk.  In summary the current
-# printk includes all preceding printk's which have no newline on the end.
-# we assume the first bad printk is the one to report.
-		if ($line =~ /\bprintk\((?!KERN_)\s*"/) {
-			my $ok = 0;
-			for (my $ln = $linenr - 1; $ln >= $first_line; $ln--) {
-				#print "CHECK<$lines[$ln - 1]\n";
-				# we have a preceding printk if it ends
-				# with "\n" ignore it, else it is to blame
-				if ($lines[$ln - 1] =~ m{\bprintk\(}) {
-					if ($rawlines[$ln - 1] !~ m{\\n"}) {
-						$ok = 1;
-					}
-					last;
-				}
-			}
-			if ($ok == 0) {
-				WARN("PRINTK_WITHOUT_KERN_LEVEL",
-				     "printk() should include KERN_ facility level\n" . $herecurr);
-			}
+# printk should use KERN_* levels
+		if ($line =~ /\bprintk\s*\(\s*(?!KERN_[A-Z]+\b)/) {
+			WARN("PRINTK_WITHOUT_KERN_LEVEL",
+			     "printk() should include KERN_<LEVEL> facility level\n" . $herecurr);
 		}
 
 		if ($line =~ /\bprintk\s*\(\s*KERN_([A-Z]+)/) {
@@ -3595,10 +3963,12 @@ sub process {
 
 # function brace can't be on same line, except for #defines of do while,
 # or if closed on same line
-		if (($line=~/$Type\s*$Ident\(.*\).*\s*{/) and
-		    !($line=~/\#\s*define.*do\s\{/) and !($line=~/}/)) {
+		if ($^V && $^V ge 5.10.0 &&
+		    $sline =~ /$Type\s*$Ident\s*$balanced_parens\s*\{/ &&
+		    $sline !~ /\#\s*define\b.*do\s*\{/ &&
+		    $sline !~ /}/) {
 			if (ERROR("OPEN_BRACE",
-				  "open brace '{' following function declarations go on the next line\n" . $herecurr) &&
+				  "open brace '{' following function definitions go on the next line\n" . $herecurr) &&
 			    $fix) {
 				fix_delete_line($fixlinenr, $rawline);
 				my $fixed_line = $rawline;
@@ -3624,7 +3994,7 @@ sub process {
 				my $fixedline = rtrim($prevrawline) . " {";
 				fix_insert_line($fixlinenr, $fixedline);
 				$fixedline = $rawline;
-				$fixedline =~ s/^(.\s*){\s*/$1\t/;
+				$fixedline =~ s/^(.\s*)\{\s*/$1\t/;
 				if ($fixedline !~ /^\+\s*$/) {
 					fix_insert_line($fixlinenr, $fixedline);
 				}
@@ -3719,7 +4089,7 @@ sub process {
 			my ($where, $prefix) = ($-[1], $1);
 			if ($prefix !~ /$Type\s+$/ &&
 			    ($where != 0 || $prefix !~ /^.\s+$/) &&
-			    $prefix !~ /[{,]\s+$/) {
+			    $prefix !~ /[{,:]\s+$/) {
 				if (ERROR("BRACKET_SPACE",
 					  "space prohibited before open square bracket '['\n" . $herecurr) &&
 				    $fix) {
@@ -4044,11 +4414,11 @@ sub process {
 
 					# messages are ERROR, but ?: are CHK
 					if ($ok == 0) {
-						my $msg_type = \&ERROR;
-						$msg_type = \&CHK if (($op eq '?:' || $op eq '?' || $op eq ':') && $ctx =~ /VxV/);
+						my $msg_level = \&ERROR;
+						$msg_level = \&CHK if (($op eq '?:' || $op eq '?' || $op eq ':') && $ctx =~ /VxV/);
 
-						if (&{$msg_type}("SPACING",
-								 "spaces required around that '$op' $at\n" . $hereptr)) {
+						if (&{$msg_level}("SPACING",
+								  "spaces required around that '$op' $at\n" . $hereptr)) {
 							$good = rtrim($fix_elements[$n]) . " " . trim($fix_elements[$n + 1]) . " ";
 							if (defined $fix_elements[$n + 2]) {
 								$fix_elements[$n + 2] =~ s/^\s+//;
@@ -4108,12 +4478,12 @@ sub process {
 ## 		}
 
 #need space before brace following if, while, etc
-		if (($line =~ /\(.*\)\{/ && $line !~ /\($Type\){/) ||
+		if (($line =~ /\(.*\)\{/ && $line !~ /\($Type\)\{/) ||
 		    $line =~ /do\{/) {
 			if (ERROR("SPACING",
 				  "space required before the open brace '{'\n" . $herecurr) &&
 			    $fix) {
-				$fixed[$fixlinenr] =~ s/^(\+.*(?:do|\))){/$1 {/;
+				$fixed[$fixlinenr] =~ s/^(\+.*(?:do|\)))\{/$1 {/;
 			}
 		}
 
@@ -4201,6 +4571,32 @@ sub process {
 			}
 		}
 
+# check for unnecessary parentheses around comparisons in if uses
+# when !drivers/staging or command-line uses --strict
+		if (($realfile !~ m@^(?:drivers/staging/)@ || $check_orig) &&
+		    $^V && $^V ge 5.10.0 && defined($stat) &&
+		    $stat =~ /(^.\s*if\s*($balanced_parens))/) {
+			my $if_stat = $1;
+			my $test = substr($2, 1, -1);
+			my $herectx;
+			while ($test =~ /(?:^|[^\w\&\!\~])+\s*\(\s*([\&\!\~]?\s*$Lval\s*(?:$Compare\s*$FuncArg)?)\s*\)/g) {
+				my $match = $1;
+				# avoid parentheses around potential macro args
+				next if ($match =~ /^\s*\w+\s*$/);
+				if (!defined($herectx)) {
+					$herectx = $here . "\n";
+					my $cnt = statement_rawlines($if_stat);
+					for (my $n = 0; $n < $cnt; $n++) {
+						my $rl = raw_line($linenr, $n);
+						$herectx .=  $rl . "\n";
+						last if $rl =~ /^[ \+].*\{/;
+					}
+				}
+				CHK("UNNECESSARY_PARENTHESES",
+				    "Unnecessary parentheses around '$match'\n" . $herectx);
+			}
+		}
+
 #goto labels aren't indented, allow a single space however
 		if ($line=~/^.\s+[A-Za-z\d_]+:(?![0-9]+)/ and
 		   !($line=~/^. [A-Za-z\d_]+:/) and !($line=~/^.\s+default:/)) {
@@ -4266,7 +4662,7 @@ sub process {
 			my $comp = $3;
 			my $to = $4;
 			my $newcomp = $comp;
-			if ($lead !~ /$Operators\s*$/ &&
+			if ($lead !~ /(?:$Operators|\.)\s*$/ &&
 			    $to !~ /^(?:Constant|[A-Z_][A-Z0-9_]*)$/ &&
 			    WARN("CONSTANT_COMPARISON",
 				 "Comparisons should place the constant on the right side of the test\n" . $herecurr) &&
@@ -4541,7 +4937,17 @@ sub process {
 			$has_flow_statement = 1 if ($ctx =~ /\b(goto|return)\b/);
 			$has_arg_concat = 1 if ($ctx =~ /\#\#/ && $ctx !~ /\#\#\s*(?:__VA_ARGS__|args)\b/);
 
-			$dstat =~ s/^.\s*\#\s*define\s+$Ident(?:\([^\)]*\))?\s*//;
+			$dstat =~ s/^.\s*\#\s*define\s+$Ident(\([^\)]*\))?\s*//;
+			my $define_args = $1;
+			my $define_stmt = $dstat;
+			my @def_args = ();
+
+			if (defined $define_args && $define_args ne "") {
+				$define_args = substr($define_args, 1, length($define_args) - 2);
+				$define_args =~ s/\s*//g;
+				@def_args = split(",", $define_args);
+			}
+
 			$dstat =~ s/$;//g;
 			$dstat =~ s/\\\n.//g;
 			$dstat =~ s/^\s*//s;
@@ -4560,6 +4966,9 @@ sub process {
 			{
 			}
 
+			# Make asm volatile uses seem like a generic function
+			$dstat =~ s/\b_*asm_*\s+_*volatile_*\b/asm_volatile/g;
+
 			my $exceptions = qr{
 				$Declare|
 				module_param_named|
@@ -4574,6 +4983,11 @@ sub process {
 				^\[
 			}x;
 			#print "REST<$rest> dstat<$dstat> ctx<$ctx>\n";
+
+			$ctx =~ s/\n*$//;
+			my $stmt_cnt = statement_rawlines($ctx);
+			my $herectx = get_stat_here($linenr, $stmt_cnt, $here);
+
 			if ($dstat ne '' &&
 			    $dstat !~ /^(?:$Ident|-?$Constant),$/ &&			# 10, // foo(),
 			    $dstat !~ /^(?:$Ident|-?$Constant);$/ &&			# foo();
@@ -4589,32 +5003,64 @@ sub process {
 			    $dstat !~ /^\(\{/ &&						# ({...
 			    $ctx !~ /^.\s*#\s*define\s+TRACE_(?:SYSTEM|INCLUDE_FILE|INCLUDE_PATH)\b/)
 			{
-				$ctx =~ s/\n*$//;
-				my $herectx = $here . "\n";
-				my $cnt = statement_rawlines($ctx);
-
-				for (my $n = 0; $n < $cnt; $n++) {
-					$herectx .= raw_line($linenr, $n) . "\n";
-				}
-
-				if ($dstat =~ /;/) {
+				if ($dstat =~ /^\s*if\b/) {
+					ERROR("MULTISTATEMENT_MACRO_USE_DO_WHILE",
+					      "Macros starting with if should be enclosed by a do - while loop to avoid possible if/else logic defects\n" . "$herectx");
+				} elsif ($dstat =~ /;/) {
 					ERROR("MULTISTATEMENT_MACRO_USE_DO_WHILE",
 					      "Macros with multiple statements should be enclosed in a do - while loop\n" . "$herectx");
 				} else {
 					ERROR("COMPLEX_MACRO",
 					      "Macros with complex values should be enclosed in parentheses\n" . "$herectx");
 				}
+
+			}
+
+			# Make $define_stmt single line, comment-free, etc
+			my @stmt_array = split('\n', $define_stmt);
+			my $first = 1;
+			$define_stmt = "";
+			foreach my $l (@stmt_array) {
+				$l =~ s/\\$//;
+				if ($first) {
+					$define_stmt = $l;
+					$first = 0;
+				} elsif ($l =~ /^[\+ ]/) {
+					$define_stmt .= substr($l, 1);
+				}
+			}
+			$define_stmt =~ s/$;//g;
+			$define_stmt =~ s/\s+/ /g;
+			$define_stmt = trim($define_stmt);
+
+# check if any macro arguments are reused (ignore '...' and 'type')
+			foreach my $arg (@def_args) {
+			        next if ($arg =~ /\.\.\./);
+			        next if ($arg =~ /^type$/i);
+				my $tmp_stmt = $define_stmt;
+				$tmp_stmt =~ s/\b(typeof|__typeof__|__builtin\w+|typecheck\s*\(\s*$Type\s*,|\#+)\s*\(*\s*$arg\s*\)*\b//g;
+				$tmp_stmt =~ s/\#+\s*$arg\b//g;
+				$tmp_stmt =~ s/\b$arg\s*\#\#//g;
+				my $use_cnt = () = $tmp_stmt =~ /\b$arg\b/g;
+				if ($use_cnt > 1) {
+					CHK("MACRO_ARG_REUSE",
+					    "Macro argument reuse '$arg' - possible side-effects?\n" . "$herectx");
+				    }
+# check if any macro arguments may have other precedence issues
+				if ($tmp_stmt =~ m/($Operators)?\s*\b$arg\b\s*($Operators)?/m &&
+				    ((defined($1) && $1 ne ',') ||
+				     (defined($2) && $2 ne ','))) {
+					CHK("MACRO_ARG_PRECEDENCE",
+					    "Macro argument '$arg' may be better as '($arg)' to avoid precedence issues\n" . "$herectx");
+				}
 			}
 
 # check for macros with flow control, but without ## concatenation
 # ## concatenation is commonly a macro that defines a function so ignore those
 			if ($has_flow_statement && !$has_arg_concat) {
-				my $herectx = $here . "\n";
 				my $cnt = statement_rawlines($ctx);
+				my $herectx = get_stat_here($linenr, $cnt, $here);
 
-				for (my $n = 0; $n < $cnt; $n++) {
-					$herectx .= raw_line($linenr, $n) . "\n";
-				}
 				WARN("MACRO_WITH_FLOW_CONTROL",
 				     "Macros with flow control statements should be avoided\n" . "$herectx");
 			}
@@ -4654,11 +5100,7 @@ sub process {
 
 				$ctx =~ s/\n*$//;
 				my $cnt = statement_rawlines($ctx);
-				my $herectx = $here . "\n";
-
-				for (my $n = 0; $n < $cnt; $n++) {
-					$herectx .= raw_line($linenr, $n) . "\n";
-				}
+				my $herectx = get_stat_here($linenr, $cnt, $here);
 
 				if (($stmts =~ tr/;/;/) == 1 &&
 				    $stmts !~ /^\s*(if|while|for|switch)\b/) {
@@ -4672,11 +5114,7 @@ sub process {
 			} elsif ($dstat =~ /^\+\s*#\s*define\s+$Ident.*;\s*$/) {
 				$ctx =~ s/\n*$//;
 				my $cnt = statement_rawlines($ctx);
-				my $herectx = $here . "\n";
-
-				for (my $n = 0; $n < $cnt; $n++) {
-					$herectx .= raw_line($linenr, $n) . "\n";
-				}
+				my $herectx = get_stat_here($linenr, $cnt, $here);
 
 				WARN("TRAILING_SEMICOLON",
 				     "macros should not use a trailing semicolon\n" . "$herectx");
@@ -4799,18 +5237,20 @@ sub process {
 				}
 			}
 			if ($level == 0 && $block =~ /^\s*\{/ && !$allowed) {
-				my $herectx = $here . "\n";
 				my $cnt = statement_rawlines($block);
-
-				for (my $n = 0; $n < $cnt; $n++) {
-					$herectx .= raw_line($linenr, $n) . "\n";
-				}
+				my $herectx = get_stat_here($linenr, $cnt, $here);
 
 				WARN("BRACES",
 				     "braces {} are not necessary for single statement blocks\n" . $herectx);
 			}
 		}
 
+# check for single line unbalanced braces
+		if ($sline =~ /^.\s*\}\s*else\s*$/ ||
+		    $sline =~ /^.\s*else\s*\{\s*$/) {
+			CHK("BRACES", "Unbalanced braces around else statement\n" . $herecurr);
+		}
+
 # check for unnecessary blank lines around braces
 		if (($line =~ /^.\s*}\s*$/ && $prevrawline =~ /^.\s*$/)) {
 			if (CHK("BRACES",
@@ -4831,7 +5271,7 @@ sub process {
 		my $asm_volatile = qr{\b(__asm__|asm)\s+(__volatile__|volatile)\b};
 		if ($line =~ /\bvolatile\b/ && $line !~ /$asm_volatile/) {
 			WARN("VOLATILE",
-			     "Use of volatile is usually wrong: see Documentation/volatile-considered-harmful.txt\n" . $herecurr);
+			     "Use of volatile is usually wrong: see Documentation/process/volatile-considered-harmful.rst\n" . $herecurr);
 		}
 
 # Check for user-visible strings broken across lines, which breaks the ability
@@ -4873,6 +5313,18 @@ sub process {
 			     "break quoted strings at a space character\n" . $hereprev);
 		}
 
+# check for an embedded function name in a string when the function is known
+# This does not work very well for -f --file checking as it depends on patch
+# context providing the function name or a single line form for in-file
+# function declarations
+		if ($line =~ /^\+.*$String/ &&
+		    defined($context_function) &&
+		    get_quoted_string($line, $rawline) =~ /\b$context_function\b/ &&
+		    length(get_quoted_string($line, $rawline)) != (length($context_function) + 2)) {
+			WARN("EMBEDDED_FUNCTION_NAME",
+			     "Prefer using '\"%s...\", __func__' to using '$context_function', this function's name, in a string\n" . $herecurr);
+		}
+
 # check for spaces before a quoted newline
 		if ($rawline =~ /^.*\".*\s\\n/) {
 			if (WARN("QUOTED_WHITESPACE_BEFORE_NEWLINE",
@@ -4895,24 +5347,33 @@ sub process {
 			     "Consecutive strings are generally better as a single string\n" . $herecurr);
 		}
 
-# check for %L{u,d,i} and 0x%[udi] in strings
-		my $string;
+# check for non-standard and hex prefixed decimal printf formats
+		my $show_L = 1;	#don't show the same defect twice
+		my $show_Z = 1;
 		while ($line =~ /(?:^|")([X\t]*)(?:"|$)/g) {
-			$string = substr($rawline, $-[1], $+[1] - $-[1]);
+			my $string = substr($rawline, $-[1], $+[1] - $-[1]);
 			$string =~ s/%%/__/g;
-			if ($string =~ /(?<!%)%[\*\d\.\$]*L[udi]/) {
+			# check for %L
+			if ($show_L && $string =~ /%[\*\d\.\$]*L([diouxX])/) {
 				WARN("PRINTF_L",
-				     "\%Ld/%Lu are not-standard C, use %lld/%llu\n" . $herecurr);
-				last;
-			}
-			if ($string =~ /0x%[\*\d\.\$\Llzth]*[udi]/) {
-				ERROR("PRINTF_0xDECIMAL",
+				     "\%L$1 is non-standard C, use %ll$1\n" . $herecurr);
+				$show_L = 0;
+			}
+			# check for %Z
+			if ($show_Z && $string =~ /%[\*\d\.\$]*Z([diouxX])/) {
+				WARN("PRINTF_Z",
+				     "%Z$1 is non-standard C, use %z$1\n" . $herecurr);
+				$show_Z = 0;
+			}
+			# check for 0x<decimal>
+			if ($string =~ /0x%[\*\d\.\$\Llzth]*[diou]/) {
+				ERROR("PRINTF_0XDECIMAL",
 				      "Prefixing 0x with decimal output is defective\n" . $herecurr);
 			}
 		}
 
 # check for line continuations in quoted strings with odd counts of "
-		if ($rawline =~ /\\$/ && $rawline =~ tr/"/"/ % 2) {
+		if ($rawline =~ /\\$/ && $sline =~ tr/"/"/ % 2) {
 			WARN("LINE_CONTINUATIONS",
 			     "Avoid line continuations in quoted strings\n" . $herecurr);
 		}
@@ -4968,7 +5429,7 @@ sub process {
 			my ($s, $c) = ctx_statement_block($linenr - 3, $realcnt, 0);
 #			print("line: <$line>\nprevline: <$prevline>\ns: <$s>\nc: <$c>\n\n\n");
 
-			if ($c =~ /(?:^|\n)[ \+]\s*(?:$Type\s*)?\Q$testval\E\s*=\s*(?:\([^\)]*\)\s*)?\s*(?:devm_)?(?:[kv][czm]alloc(?:_node|_array)?\b|kstrdup|(?:dev_)?alloc_skb)/) {
+			if ($s =~ /(?:^|\n)[ \+]\s*(?:$Type\s*)?\Q$testval\E\s*=\s*(?:\([^\)]*\)\s*)?\s*(?:devm_)?(?:[kv][czm]alloc(?:_node|_array)?\b|kstrdup|kmemdup|(?:dev_)?alloc_skb)/) {
 				WARN("OOM_MESSAGE",
 				     "Possible unnecessary 'out of memory' message\n" . $hereprev);
 			}
@@ -4985,6 +5446,12 @@ sub process {
 			}
 		}
 
+# check for logging continuations
+		if ($line =~ /\bprintk\s*\(\s*KERN_CONT\b|\bpr_cont\s*\(/) {
+			WARN("LOGGING_CONTINUATION",
+			     "Avoid logging continuation uses where feasible\n" . $herecurr);
+		}
+
 # check for mask then right shift without a parentheses
 		if ($^V && $^V ge 5.10.0 &&
 		    $line =~ /$LvalOrFunc\s*\&\s*($LvalOrFunc)\s*>>/ &&
@@ -5185,21 +5652,10 @@ sub process {
 			}
 		}
 
-# Check for expedited grace periods that interrupt non-idle non-nohz
-# online CPUs.  These expedited can therefore degrade real-time response
-# if used carelessly, and should be avoided where not absolutely
-# needed.  It is always OK to use synchronize_rcu_expedited() and
-# synchronize_sched_expedited() at boot time (before real-time applications
-# start) and in error situations where real-time response is compromised in
-# any case.  Note that synchronize_srcu_expedited() does -not- interrupt
-# other CPUs, so don't warn on uses of synchronize_srcu_expedited().
-# Of course, nothing comes for free, and srcu_read_lock() and
-# srcu_read_unlock() do contain full memory barriers in payment for
-# synchronize_srcu_expedited() non-interruption properties.
-		if ($line =~ /\b(synchronize_rcu_expedited|synchronize_sched_expedited)\(/) {
-			WARN("EXPEDITED_RCU_GRACE_PERIOD",
-			     "expedited RCU grace periods should be avoided where they can degrade real-time response\n" . $herecurr);
-
+# check for smp_read_barrier_depends and read_barrier_depends
+		if (!$file && $line =~ /\b(smp_|)read_barrier_depends\s*\(/) {
+			WARN("READ_BARRIER_DEPENDS",
+			     "$1read_barrier_depends should only be used in READ_ONCE or DEC Alpha code\n" . $herecurr);
 		}
 
 # check of hardware specific defines
@@ -5208,10 +5664,18 @@ sub process {
 			    "architecture specific defines should be avoided\n" .  $herecurr);
 		}
 
+# check that the storage class is not after a type
+		if ($line =~ /\b($Type)\s+($Storage)\b/) {
+			WARN("STORAGE_CLASS",
+			     "storage class '$2' should be located before type '$1'\n" . $herecurr);
+		}
 # Check that the storage class is at the beginning of a declaration
-		if ($line =~ /\b$Storage\b/ && $line !~ /^.\s*$Storage\b/) {
+		if ($line =~ /\b$Storage\b/ &&
+		    $line !~ /^.\s*$Storage/ &&
+		    $line =~ /^.\s*(.+?)\$Storage\s/ &&
+		    $1 !~ /[\,\)]\s*$/) {
 			WARN("STORAGE_CLASS",
-			     "storage class should be at the beginning of the declaration\n" . $herecurr)
+			     "storage class should be at the beginning of the declaration\n" . $herecurr);
 		}
 
 # check the location of the inline attribute, that it is between
@@ -5277,8 +5741,9 @@ sub process {
 			      "Using weak declarations can have unintended link defects\n" . $herecurr);
 		}
 
-# check for c99 types like uint8_t used outside of uapi/
+# check for c99 types like uint8_t used outside of uapi/ and tools/
 		if ($realfile !~ m@\binclude/uapi/@ &&
+		    $realfile !~ m@\btools/@ &&
 		    $line =~ /\b($Declare)\s*$Ident\s*[=;,\[]/) {
 			my $type = $1;
 			if ($type =~ /\b($typeC99Typedefs)\b/) {
@@ -5349,6 +5814,53 @@ sub process {
 			}
 		}
 
+# check for vsprintf extension %p<foo> misuses
+		if ($^V && $^V ge 5.10.0 &&
+		    defined $stat &&
+		    $stat =~ /^\+(?![^\{]*\{\s*).*\b(\w+)\s*\(.*$String\s*,/s &&
+		    $1 !~ /^_*volatile_*$/) {
+			my $specifier;
+			my $extension;
+			my $bad_specifier = "";
+			my $stat_real;
+
+			my $lc = $stat =~ tr@\n@@;
+			$lc = $lc + $linenr;
+		        for (my $count = $linenr; $count <= $lc; $count++) {
+				my $fmt = get_quoted_string($lines[$count - 1], raw_line($count, 0));
+				$fmt =~ s/%%//g;
+
+				while ($fmt =~ /(\%[\*\d\.]*p(\w))/g) {
+					$specifier = $1;
+					$extension = $2;
+					if ($extension !~ /[SsBKRraEhMmIiUDdgVCbGNOx]/) {
+						$bad_specifier = $specifier;
+						last;
+					}
+					if ($extension eq "x" && !defined($stat_real)) {
+						if (!defined($stat_real)) {
+							$stat_real = get_stat_real($linenr, $lc);
+						}
+						WARN("VSPRINTF_SPECIFIER_PX",
+						     "Using vsprintf specifier '\%px' potentially exposes the kernel memory layout, if you don't really need the address please consider using '\%p'.\n" . "$here\n$stat_real\n");
+					}
+				}
+				if ($bad_specifier ne "") {
+					my $stat_real = get_stat_real($linenr, $lc);
+					my $ext_type = "Invalid";
+					my $use = "";
+					if ($bad_specifier =~ /p[Ff]/) {
+						$ext_type = "Deprecated";
+						$use = " - use %pS instead";
+						$use =~ s/pS/ps/ if ($bad_specifier =~ /pf/);
+					}
+
+					WARN("VSPRINTF_POINTER_EXTENSION",
+					     "$ext_type vsprintf pointer extension '$bad_specifier'$use\n" . "$here\n$stat_real\n");
+				}
+			}
+		}
+
 # Check for misused memsets
 		if ($^V && $^V ge 5.10.0 &&
 		    defined $stat &&
@@ -5368,46 +5880,46 @@ sub process {
 		}
 
 # Check for memcpy(foo, bar, ETH_ALEN) that could be ether_addr_copy(foo, bar)
-		if ($^V && $^V ge 5.10.0 &&
-		    defined $stat &&
-		    $stat =~ /^\+(?:.*?)\bmemcpy\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*\,\s*ETH_ALEN\s*\)/) {
-			if (WARN("PREFER_ETHER_ADDR_COPY",
-				 "Prefer ether_addr_copy() over memcpy() if the Ethernet addresses are __aligned(2)\n" . "$here\n$stat\n") &&
-			    $fix) {
-				$fixed[$fixlinenr] =~ s/\bmemcpy\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*\,\s*ETH_ALEN\s*\)/ether_addr_copy($2, $7)/;
-			}
-		}
+#		if ($^V && $^V ge 5.10.0 &&
+#		    defined $stat &&
+#		    $stat =~ /^\+(?:.*?)\bmemcpy\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*\,\s*ETH_ALEN\s*\)/) {
+#			if (WARN("PREFER_ETHER_ADDR_COPY",
+#				 "Prefer ether_addr_copy() over memcpy() if the Ethernet addresses are __aligned(2)\n" . "$here\n$stat\n") &&
+#			    $fix) {
+#				$fixed[$fixlinenr] =~ s/\bmemcpy\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*\,\s*ETH_ALEN\s*\)/ether_addr_copy($2, $7)/;
+#			}
+#		}
 
 # Check for memcmp(foo, bar, ETH_ALEN) that could be ether_addr_equal*(foo, bar)
-		if ($^V && $^V ge 5.10.0 &&
-		    defined $stat &&
-		    $stat =~ /^\+(?:.*?)\bmemcmp\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*\,\s*ETH_ALEN\s*\)/) {
-			WARN("PREFER_ETHER_ADDR_EQUAL",
-			     "Prefer ether_addr_equal() or ether_addr_equal_unaligned() over memcmp()\n" . "$here\n$stat\n")
-		}
+#		if ($^V && $^V ge 5.10.0 &&
+#		    defined $stat &&
+#		    $stat =~ /^\+(?:.*?)\bmemcmp\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*\,\s*ETH_ALEN\s*\)/) {
+#			WARN("PREFER_ETHER_ADDR_EQUAL",
+#			     "Prefer ether_addr_equal() or ether_addr_equal_unaligned() over memcmp()\n" . "$here\n$stat\n")
+#		}
 
 # check for memset(foo, 0x0, ETH_ALEN) that could be eth_zero_addr
 # check for memset(foo, 0xFF, ETH_ALEN) that could be eth_broadcast_addr
-		if ($^V && $^V ge 5.10.0 &&
-		    defined $stat &&
-		    $stat =~ /^\+(?:.*?)\bmemset\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*\,\s*ETH_ALEN\s*\)/) {
-
-			my $ms_val = $7;
-
-			if ($ms_val =~ /^(?:0x|)0+$/i) {
-				if (WARN("PREFER_ETH_ZERO_ADDR",
-					 "Prefer eth_zero_addr over memset()\n" . "$here\n$stat\n") &&
-				    $fix) {
-					$fixed[$fixlinenr] =~ s/\bmemset\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*,\s*ETH_ALEN\s*\)/eth_zero_addr($2)/;
-				}
-			} elsif ($ms_val =~ /^(?:0xff|255)$/i) {
-				if (WARN("PREFER_ETH_BROADCAST_ADDR",
-					 "Prefer eth_broadcast_addr() over memset()\n" . "$here\n$stat\n") &&
-				    $fix) {
-					$fixed[$fixlinenr] =~ s/\bmemset\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*,\s*ETH_ALEN\s*\)/eth_broadcast_addr($2)/;
-				}
-			}
-		}
+#		if ($^V && $^V ge 5.10.0 &&
+#		    defined $stat &&
+#		    $stat =~ /^\+(?:.*?)\bmemset\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*\,\s*ETH_ALEN\s*\)/) {
+#
+#			my $ms_val = $7;
+#
+#			if ($ms_val =~ /^(?:0x|)0+$/i) {
+#				if (WARN("PREFER_ETH_ZERO_ADDR",
+#					 "Prefer eth_zero_addr over memset()\n" . "$here\n$stat\n") &&
+#				    $fix) {
+#					$fixed[$fixlinenr] =~ s/\bmemset\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*,\s*ETH_ALEN\s*\)/eth_zero_addr($2)/;
+#				}
+#			} elsif ($ms_val =~ /^(?:0xff|255)$/i) {
+#				if (WARN("PREFER_ETH_BROADCAST_ADDR",
+#					 "Prefer eth_broadcast_addr() over memset()\n" . "$here\n$stat\n") &&
+#				    $fix) {
+#					$fixed[$fixlinenr] =~ s/\bmemset\s*\(\s*$FuncArg\s*,\s*$FuncArg\s*,\s*ETH_ALEN\s*\)/eth_broadcast_addr($2)/;
+#				}
+#			}
+#		}
 
 # typecasts on min/max could be min_t/max_t
 		if ($^V && $^V ge 5.10.0 &&
@@ -5458,10 +5970,7 @@ sub process {
 		     $stat !~ /(?:$Compare)\s*\bsscanf\s*$balanced_parens/)) {
 			my $lc = $stat =~ tr@\n@@;
 			$lc = $lc + $linenr;
-			my $stat_real = raw_line($linenr, 0);
-		        for (my $count = $linenr + 1; $count <= $lc; $count++) {
-				$stat_real = $stat_real . "\n" . raw_line($count, 0);
-			}
+			my $stat_real = get_stat_real($linenr, $lc);
 			WARN("NAKED_SSCANF",
 			     "unchecked sscanf return value\n" . "$here\n$stat_real\n");
 		}
@@ -5472,10 +5981,7 @@ sub process {
 		    $line =~ /\bsscanf\b/) {
 			my $lc = $stat =~ tr@\n@@;
 			$lc = $lc + $linenr;
-			my $stat_real = raw_line($linenr, 0);
-		        for (my $count = $linenr + 1; $count <= $lc; $count++) {
-				$stat_real = $stat_real . "\n" . raw_line($count, 0);
-			}
+			my $stat_real = get_stat_real($linenr, $lc);
 			if ($stat_real =~ /\bsscanf\b\s*\(\s*$FuncArg\s*,\s*("[^"]+")/) {
 				my $format = $6;
 				my $count = $format =~ tr@%@%@;
@@ -5527,13 +6033,50 @@ sub process {
 			     "externs should be avoided in .c files\n" .  $herecurr);
 		}
 
+# check for function declarations that have arguments without identifier names
+		if (defined $stat &&
+		    $stat =~ /^.\s*(?:extern\s+)?$Type\s*(?:$Ident|\(\s*\*\s*$Ident\s*\))\s*\(\s*([^{]+)\s*\)\s*;/s &&
+		    $1 ne "void") {
+			my $args = trim($1);
+			while ($args =~ m/\s*($Type\s*(?:$Ident|\(\s*\*\s*$Ident?\s*\)\s*$balanced_parens)?)/g) {
+				my $arg = trim($1);
+				if ($arg =~ /^$Type$/ && $arg !~ /enum\s+$Ident$/) {
+					WARN("FUNCTION_ARGUMENTS",
+					     "function definition argument '$arg' should also have an identifier name\n" . $herecurr);
+				}
+			}
+		}
+
+# check for function definitions
+		if ($^V && $^V ge 5.10.0 &&
+		    defined $stat &&
+		    $stat =~ /^.\s*(?:$Storage\s+)?$Type\s*($Ident)\s*$balanced_parens\s*{/s) {
+			$context_function = $1;
+
+# check for multiline function definition with misplaced open brace
+			my $ok = 0;
+			my $cnt = statement_rawlines($stat);
+			my $herectx = $here . "\n";
+			for (my $n = 0; $n < $cnt; $n++) {
+				my $rl = raw_line($linenr, $n);
+				$herectx .=  $rl . "\n";
+				$ok = 1 if ($rl =~ /^[ \+]\{/);
+				$ok = 1 if ($rl =~ /\{/ && $n == 0);
+				last if $rl =~ /^[ \+].*\{/;
+			}
+			if (!$ok) {
+				ERROR("OPEN_BRACE",
+				      "open brace '{' following function definitions go on the next line\n" . $herectx);
+			}
+		}
+
 # checks for new __setup's
 		if ($rawline =~ /\b__setup\("([^"]*)"/) {
 			my $name = $1;
 
 			if (!grep(/$name/, @setup_docs)) {
 				CHK("UNDOCUMENTED_SETUP",
-				    "__setup appears un-documented -- check Documentation/kernel-parameters.txt\n" . $herecurr);
+				    "__setup appears un-documented -- check Documentation/admin-guide/kernel-parameters.rst\n" . $herecurr);
 			}
 		}
 
@@ -5553,7 +6096,8 @@ sub process {
 
 # check for k[mz]alloc with multiplies that could be kmalloc_array/kcalloc
 		if ($^V && $^V ge 5.10.0 &&
-		    $line =~ /\b($Lval)\s*\=\s*(?:$balanced_parens)?\s*(k[mz]alloc)\s*\(\s*($FuncArg)\s*\*\s*($FuncArg)\s*,/) {
+		    defined $stat &&
+		    $stat =~ /^\+\s*($Lval)\s*\=\s*(?:$balanced_parens)?\s*(k[mz]alloc)\s*\(\s*($FuncArg)\s*\*\s*($FuncArg)\s*,/) {
 			my $oldfunc = $3;
 			my $a1 = $4;
 			my $a2 = $10;
@@ -5567,11 +6111,14 @@ sub process {
 			}
 			if ($r1 !~ /^sizeof\b/ && $r2 =~ /^sizeof\s*\S/ &&
 			    !($r1 =~ /^$Constant$/ || $r1 =~ /^[A-Z_][A-Z0-9_]*$/)) {
+				my $cnt = statement_rawlines($stat);
+				my $herectx = get_stat_here($linenr, $cnt, $here);
+
 				if (WARN("ALLOC_WITH_MULTIPLY",
-					 "Prefer $newfunc over $oldfunc with multiply\n" . $herecurr) &&
+					 "Prefer $newfunc over $oldfunc with multiply\n" . $herectx) &&
+				    $cnt == 1 &&
 				    $fix) {
 					$fixed[$fixlinenr] =~ s/\b($Lval)\s*\=\s*(?:$balanced_parens)?\s*(k[mz]alloc)\s*\(\s*($FuncArg)\s*\*\s*($FuncArg)/$1 . ' = ' . "$newfunc(" . trim($r1) . ', ' . trim($r2)/e;
-
 				}
 			}
 		}
@@ -5598,8 +6145,9 @@ sub process {
 			}
 		}
 
-# check for #defines like: 1 << <digit> that could be BIT(digit)
-		if ($line =~ /#\s*define\s+\w+\s+\(?\s*1\s*([ulUL]*)\s*\<\<\s*(?:\d+|$Ident)\s*\)?/) {
+# check for #defines like: 1 << <digit> that could be BIT(digit), it is not exported to uapi
+		if ($realfile !~ m@^include/uapi/@ &&
+		    $line =~ /#\s*define\s+\w+\s+\(?\s*1\s*([ulUL]*)\s*\<\<\s*(?:\d+|$Ident)\s*\)?/) {
 			my $ull = "";
 			$ull = "_ULL" if (defined($1) && $1 =~ /ll/i);
 			if (CHK("BIT_MACRO",
@@ -5609,6 +6157,16 @@ sub process {
 			}
 		}
 
+# check for #if defined CONFIG_<FOO> || defined CONFIG_<FOO>_MODULE
+		if ($line =~ /^\+\s*#\s*if\s+defined(?:\s*\(?\s*|\s+)(CONFIG_[A-Z_]+)\s*\)?\s*\|\|\s*defined(?:\s*\(?\s*|\s+)\1_MODULE\s*\)?\s*$/) {
+			my $config = $1;
+			if (WARN("PREFER_IS_ENABLED",
+				 "Prefer IS_ENABLED(<FOO>) to CONFIG_<FOO> || CONFIG_<FOO>_MODULE\n" . $herecurr) &&
+			    $fix) {
+				$fixed[$fixlinenr] = "\+#if IS_ENABLED($config)";
+			}
+		}
+
 # check for case / default statements not preceded by break/fallthrough/switch
 		if ($line =~ /^.\s*(?:case\s+(?:$Ident|$Constant)\s*|default):/) {
 			my $has_break = 0;
@@ -5626,11 +6184,11 @@ sub process {
 				next if ($fline =~ /^.[\s$;]*$/);
 				$has_statement = 1;
 				$count++;
-				$has_break = 1 if ($fline =~ /\bswitch\b|\b(?:break\s*;[\s$;]*$|return\b|goto\b|continue\b)/);
+				$has_break = 1 if ($fline =~ /\bswitch\b|\b(?:break\s*;[\s$;]*$|exit\s*\(\b|return\b|goto\b|continue\b)/);
 			}
 			if (!$has_break && $has_statement) {
 				WARN("MISSING_BREAK",
-				     "Possible switch case/default not preceeded by break or fallthrough comment\n" . $herecurr);
+				     "Possible switch case/default not preceded by break or fallthrough comment\n" . $herecurr);
 			}
 		}
 
@@ -5638,12 +6196,9 @@ sub process {
 		if ($^V && $^V ge 5.10.0 &&
 		    defined $stat &&
 		    $stat =~ /^\+[$;\s]*(?:case[$;\s]+\w+[$;\s]*:[$;\s]*|)*[$;\s]*\bdefault[$;\s]*:[$;\s]*;/g) {
-			my $ctx = '';
-			my $herectx = $here . "\n";
 			my $cnt = statement_rawlines($stat);
-			for (my $n = 0; $n < $cnt; $n++) {
-				$herectx .= raw_line($linenr, $n) . "\n";
-			}
+			my $herectx = get_stat_here($linenr, $cnt, $here);
+
 			WARN("DEFAULT_NO_BREAK",
 			     "switch default: should use break\n" . $herectx);
 		}
@@ -5696,6 +6251,12 @@ sub process {
 			}
 		}
 
+# check for bool bitfields
+		if ($sline =~ /^.\s+bool\s*$Ident\s*:\s*\d+\s*;/) {
+			WARN("BOOL_BITFIELD",
+			     "Avoid using bool as bitfield.  Prefer bool bitfields as unsigned int or u<8|16|32>\n" . $herecurr);
+		}
+
 # check for semaphores initialized locked
 		if ($line =~ /^.\s*sema_init.+,\W?0\W?\)/) {
 			WARN("CONSIDER_COMPLETION",
@@ -5715,51 +6276,11 @@ sub process {
 		}
 
 # check for various structs that are normally const (ops, kgdb, device_tree)
-		my $const_structs = qr{
-				acpi_dock_ops|
-				address_space_operations|
-				backlight_ops|
-				block_device_operations|
-				dentry_operations|
-				dev_pm_ops|
-				dma_map_ops|
-				extent_io_ops|
-				file_lock_operations|
-				file_operations|
-				hv_ops|
-				ide_dma_ops|
-				intel_dvo_dev_ops|
-				item_operations|
-				iwl_ops|
-				kgdb_arch|
-				kgdb_io|
-				kset_uevent_ops|
-				lock_manager_operations|
-				microcode_ops|
-				mtrr_ops|
-				neigh_ops|
-				nlmsvc_binding|
-				of_device_id|
-				pci_raw_ops|
-				pipe_buf_operations|
-				platform_hibernation_ops|
-				platform_suspend_ops|
-				proto_ops|
-				rpc_pipe_ops|
-				seq_operations|
-				snd_ac97_build_ops|
-				soc_pcmcia_socket_ops|
-				stacktrace_ops|
-				sysfs_ops|
-				tty_operations|
-				uart_ops|
-				usb_mon_operations|
-				wd_ops}x;
+# and avoid what seem like struct definitions 'struct foo {'
 		if ($line !~ /\bconst\b/ &&
-		    $line =~ /\bstruct\s+($const_structs)\b/) {
+		    $line =~ /\bstruct\s+($const_structs)\b(?!\s*\{)/) {
 			WARN("CONST_STRUCT",
-			     "struct $1 should normally be const\n" .
-				$herecurr);
+			     "struct $1 should normally be const\n" . $herecurr);
 		}
 
 # use of NR_CPUS is usually wrong
@@ -5799,6 +6320,12 @@ sub process {
 			}
 		}
 
+# check for mutex_trylock_recursive usage
+		if ($line =~ /mutex_trylock_recursive/) {
+			ERROR("LOCKING",
+			      "recursive locking is bad, do not use this ever.\n" . $herecurr);
+		}
+
 # check for lockdep_set_novalidate_class
 		if ($line =~ /^.\s*lockdep_set_novalidate_class\s*\(/ ||
 		    $line =~ /__lockdep_no_validate__\s*\)/ ) {
@@ -5816,37 +6343,114 @@ sub process {
 			     "Exporting world writable files is usually an error. Consider more restrictive permissions.\n" . $herecurr);
 		}
 
+# check for DEVICE_ATTR uses that could be DEVICE_ATTR_<FOO>
+# and whether or not function naming is typical and if
+# DEVICE_ATTR permissions uses are unusual too
+		if ($^V && $^V ge 5.10.0 &&
+		    defined $stat &&
+		    $stat =~ /\bDEVICE_ATTR\s*\(\s*(\w+)\s*,\s*\(?\s*(\s*(?:${multi_mode_perms_string_search}|0[0-7]{3,3})\s*)\s*\)?\s*,\s*(\w+)\s*,\s*(\w+)\s*\)/) {
+			my $var = $1;
+			my $perms = $2;
+			my $show = $3;
+			my $store = $4;
+			my $octal_perms = perms_to_octal($perms);
+			if ($show =~ /^${var}_show$/ &&
+			    $store =~ /^${var}_store$/ &&
+			    $octal_perms eq "0644") {
+				if (WARN("DEVICE_ATTR_RW",
+					 "Use DEVICE_ATTR_RW\n" . $herecurr) &&
+				    $fix) {
+					$fixed[$fixlinenr] =~ s/\bDEVICE_ATTR\s*\(\s*$var\s*,\s*\Q$perms\E\s*,\s*$show\s*,\s*$store\s*\)/DEVICE_ATTR_RW(${var})/;
+				}
+			} elsif ($show =~ /^${var}_show$/ &&
+				 $store =~ /^NULL$/ &&
+				 $octal_perms eq "0444") {
+				if (WARN("DEVICE_ATTR_RO",
+					 "Use DEVICE_ATTR_RO\n" . $herecurr) &&
+				    $fix) {
+					$fixed[$fixlinenr] =~ s/\bDEVICE_ATTR\s*\(\s*$var\s*,\s*\Q$perms\E\s*,\s*$show\s*,\s*NULL\s*\)/DEVICE_ATTR_RO(${var})/;
+				}
+			} elsif ($show =~ /^NULL$/ &&
+				 $store =~ /^${var}_store$/ &&
+				 $octal_perms eq "0200") {
+				if (WARN("DEVICE_ATTR_WO",
+					 "Use DEVICE_ATTR_WO\n" . $herecurr) &&
+				    $fix) {
+					$fixed[$fixlinenr] =~ s/\bDEVICE_ATTR\s*\(\s*$var\s*,\s*\Q$perms\E\s*,\s*NULL\s*,\s*$store\s*\)/DEVICE_ATTR_WO(${var})/;
+				}
+			} elsif ($octal_perms eq "0644" ||
+				 $octal_perms eq "0444" ||
+				 $octal_perms eq "0200") {
+				my $newshow = "$show";
+				$newshow = "${var}_show" if ($show ne "NULL" && $show ne "${var}_show");
+				my $newstore = $store;
+				$newstore = "${var}_store" if ($store ne "NULL" && $store ne "${var}_store");
+				my $rename = "";
+				if ($show ne $newshow) {
+					$rename .= " '$show' to '$newshow'";
+				}
+				if ($store ne $newstore) {
+					$rename .= " '$store' to '$newstore'";
+				}
+				WARN("DEVICE_ATTR_FUNCTIONS",
+				     "Consider renaming function(s)$rename\n" . $herecurr);
+			} else {
+				WARN("DEVICE_ATTR_PERMS",
+				     "DEVICE_ATTR unusual permissions '$perms' used\n" . $herecurr);
+			}
+		}
+
 # Mode permission misuses where it seems decimal should be octal
 # This uses a shortcut match to avoid unnecessary uses of a slow foreach loop
+# o Ignore module_param*(...) uses with a decimal 0 permission as that has a
+#   specific definition of not visible in sysfs.
+# o Ignore proc_create*(...) uses with a decimal 0 permission as that means
+#   use the default permissions
 		if ($^V && $^V ge 5.10.0 &&
+		    defined $stat &&
 		    $line =~ /$mode_perms_search/) {
 			foreach my $entry (@mode_permission_funcs) {
 				my $func = $entry->[0];
 				my $arg_pos = $entry->[1];
 
+				my $lc = $stat =~ tr@\n@@;
+				$lc = $lc + $linenr;
+				my $stat_real = get_stat_real($linenr, $lc);
+
 				my $skip_args = "";
 				if ($arg_pos > 1) {
 					$arg_pos--;
 					$skip_args = "(?:\\s*$FuncArg\\s*,\\s*){$arg_pos,$arg_pos}";
 				}
-				my $test = "\\b$func\\s*\\(${skip_args}([\\d]+)\\s*[,\\)]";
-				if ($line =~ /$test/) {
+				my $test = "\\b$func\\s*\\(${skip_args}($FuncArg(?:\\|\\s*$FuncArg)*)\\s*[,\\)]";
+				if ($stat =~ /$test/) {
 					my $val = $1;
 					$val = $6 if ($skip_args ne "");
-
-					if ($val !~ /^0$/ &&
+					if (!($func =~ /^(?:module_param|proc_create)/ && $val eq "0") &&
 					    (($val =~ /^$Int$/ && $val !~ /^$Octal$/) ||
-					     length($val) ne 4)) {
+					     ($val =~ /^$Octal$/ && length($val) ne 4))) {
 						ERROR("NON_OCTAL_PERMISSIONS",
-						      "Use 4 digit octal (0777) not decimal permissions\n" . $herecurr);
-					} elsif ($val =~ /^$Octal$/ && (oct($val) & 02)) {
+						      "Use 4 digit octal (0777) not decimal permissions\n" . "$here\n" . $stat_real);
+					}
+					if ($val =~ /^$Octal$/ && (oct($val) & 02)) {
 						ERROR("EXPORTED_WORLD_WRITABLE",
-						      "Exporting writable files is usually an error. Consider more restrictive permissions.\n" . $herecurr);
+						      "Exporting writable files is usually an error. Consider more restrictive permissions.\n" . "$here\n" . $stat_real);
 					}
 				}
 			}
 		}
 
+# check for uses of S_<PERMS> that could be octal for readability
+		while ($line =~ m{\b($multi_mode_perms_string_search)\b}g) {
+			my $oval = $1;
+			my $octal = perms_to_octal($oval);
+			if (WARN("SYMBOLIC_PERMS",
+				 "Symbolic permissions '$oval' are not preferred. Consider using octal permissions '$octal'.\n" . $herecurr) &&
+			    $fix) {
+				$fixed[$fixlinenr] =~ s/\Q$oval\E/$octal/;
+			}
+		}
+
 # validate content of MODULE_LICENSE against list from include/linux/module.h
 		if ($line =~ /\bMODULE_LICENSE\s*\(\s*($String)\s*\)/) {
 			my $extracted_string = get_quoted_string($line, $rawline);
@@ -5884,11 +6488,11 @@ sub process {
 		exit(0);
 	}
 
-	if (!$is_patch && $file !~ /cover-letter\.patch$/) {
+	if (!$is_patch && $filename !~ /cover-letter\.patch$/) {
 		ERROR("NOT_UNIFIED_DIFF",
 		      "Does not appear to be a unified-diff format patch\n");
 	}
-	if ($is_patch && $filename ne '-' && $chk_signoff && $signoff == 0) {
+	if ($is_patch && $has_commit_log && $chk_signoff && $signoff == 0) {
 		ERROR("MISSING_SIGN_OFF",
 		      "Missing Signed-off-by: line(s)\n");
 	}
@@ -5902,6 +6506,14 @@ sub process {
 	}
 
 	if ($quiet == 0) {
+		# If there were any defects found and not already fixing them
+		if (!$clean and !$fix) {
+			print << "EOM"
+
+NOTE: For some of the reported defects, checkpatch may be able to
+      mechanically convert to the typical style using --fix or --fix-inplace.
+EOM
+		}
 		# If there were whitespace errors which cleanpatch can fix
 		# then suggest that.
 		if ($rpt_cleaners) {
diff --git a/tools/checkpatch.pl-update b/tools/checkpatch.pl-update
index 1a25797..2462038 100755
--- a/tools/checkpatch.pl-update
+++ b/tools/checkpatch.pl-update
@@ -46,8 +46,9 @@ get_latest_version() {
   output=$(git ls-remote --tags "${GIT_URL}")
   # Filter out rc/signed tags.
   output=$(echo "${output}" | grep -v -e '-rc' -e '\^{}')
-  # The output should be sorted already, so grab the last entry.
-  output=$(echo "${output}" | tail -1 | cut -d/ -f3)
+  # The output might not be sorted based on versions, so do so ourselves and
+  # grab the last entry.
+  output=$(echo "${output}" | cut -d/ -f3 | sort -V | tail -1)
   # Chop the leading "v" bit.
   echo "${output#v}"
 }
diff --git a/tools/clang-format.py b/tools/clang-format.py
index 1b109e3..4a327e0 100755
--- a/tools/clang-format.py
+++ b/tools/clang-format.py
@@ -28,6 +28,9 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# We have to import our local modules after the sys.path tweak.  We can't use
+# relative imports because this is an executable program, not a module.
+# pylint: disable=wrong-import-position
 import rh.shell
 import rh.utils
 
@@ -84,7 +87,15 @@ def main(argv):
     if platform.system() == 'Windows':
       cmd = ['python.exe'] + cmd
 
-    stdout = rh.utils.run_command(cmd, capture_output=True).output
+    # Fail gracefully if clang-format itself aborts/fails.
+    try:
+        result = rh.utils.run_command(cmd, capture_output=True)
+    except rh.utils.RunCommandError as e:
+        print('clang-format failed:\n%s' % (e,), file=sys.stderr)
+        print('\nPlease report this to the clang team.', file=sys.stderr)
+        return 1
+
+    stdout = result.output
     if stdout.rstrip('\n') == 'no modified files to format':
         # This is always printed when only files that clang-format does not
         # understand were modified.
@@ -97,7 +108,15 @@ def main(argv):
 
     if diff_filenames:
         if opts.fix:
-            rh.utils.run_command(['git', 'apply'], input=stdout)
+            result = rh.utils.run_command(['git', 'apply'], input=stdout,
+                                          error_code_ok=True)
+            if result.returncode:
+                print('Error: Unable to automatically fix things.\n'
+                      '  Make sure your checkout is clean first.\n'
+                      '  If you have multiple commits, you might have to '
+                      'manually rebase your tree first.',
+                      file=sys.stderr)
+                return result.returncode
         else:
             print('The following files have formatting errors:')
             for filename in diff_filenames:
diff --git a/tools/cpplint.py b/tools/cpplint.py
index 796a4a4..c3c2ccc 100755
--- a/tools/cpplint.py
+++ b/tools/cpplint.py
@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+# pylint: skip-file
 #
 # Copyright (c) 2009 Google Inc. All rights reserved.
 #
@@ -56,7 +57,8 @@ import unicodedata
 _USAGE = """
 Syntax: cpplint.py [--verbose=#] [--output=vs7] [--filter=-x,+y,...]
                    [--counting=total|toplevel|detailed] [--root=subdir]
-                   [--linelength=digits]
+                   [--linelength=digits] [--headers=x,y,...]
+                   [--quiet]
         <file> [file] ...
 
   The style guidelines this tries to follow are those in
@@ -83,6 +85,9 @@ Syntax: cpplint.py [--verbose=#] [--output=vs7] [--filter=-x,+y,...]
     verbose=#
       Specify a number 0-5 to restrict errors to certain verbosity levels.
 
+    quiet
+      Don't print anything if no errors are found.
+
     filter=-x,+y,...
       Specify a comma-separated list of category-filters to apply: only
       error messages whose category names pass the filters will be printed.
@@ -114,12 +119,13 @@ Syntax: cpplint.py [--verbose=#] [--output=vs7] [--filter=-x,+y,...]
       ignored.
 
       Examples:
-        Assuming that src/.git exists, the header guard CPP variables for
-        src/chrome/browser/ui/browser.h are:
+        Assuming that top/src/.git exists (and cwd=top/src), the header guard
+        CPP variables for top/src/chrome/browser/ui/browser.h are:
 
         No flag => CHROME_BROWSER_UI_BROWSER_H_
         --root=chrome => BROWSER_UI_BROWSER_H_
         --root=chrome/browser => UI_BROWSER_H_
+        --root=.. => SRC_CHROME_BROWSER_UI_BROWSER_H_
 
     linelength=digits
       This is the allowed line length for the project. The default value is
@@ -134,6 +140,14 @@ Syntax: cpplint.py [--verbose=#] [--output=vs7] [--filter=-x,+y,...]
       Examples:
         --extensions=hpp,cpp
 
+    headers=x,y,...
+      The header extensions that cpplint will treat as .h in checks. Values are
+      automatically added to --extensions list.
+
+      Examples:
+        --headers=hpp,hxx
+        --headers=hpp
+
     cpplint.py supports per-directory configurations specified in CPPLINT.cfg
     files. CPPLINT.cfg file can contain a number of key=value pairs.
     Currently the following options are supported:
@@ -143,6 +157,7 @@ Syntax: cpplint.py [--verbose=#] [--output=vs7] [--filter=-x,+y,...]
       exclude_files=regex
       linelength=80
       root=subdir
+      headers=x,y,...
 
     "set noparent" option prevents cpplint from traversing directory tree
     upwards looking for more .cfg files in parent directories. This option
@@ -159,7 +174,10 @@ Syntax: cpplint.py [--verbose=#] [--output=vs7] [--filter=-x,+y,...]
     "linelength" allows to specify the allowed line length for the project.
 
     The "root" option is similar in function to the --root flag (see example
-    above).
+    above). Paths are relative to the directory of the CPPLINT.cfg.
+
+    The "headers" option is similar in function to the --headers flag
+    (see example above).
 
     CPPLINT.cfg has an effect on files in the same directory and all
     sub-directories, unless overridden by a nested configuration file.
@@ -527,6 +545,7 @@ _error_suppressions = {}
 # The root directory used for deriving header guard CPP variable.
 # This is set by --root flag.
 _root = None
+_root_debug = False
 
 # The allowed line length of files.
 # This is set by --linelength flag.
@@ -536,10 +555,25 @@ _line_length = 80
 # This is set by --extensions flag.
 _valid_extensions = set(['cc', 'h', 'cpp', 'cu', 'cuh'])
 
+# Treat all headers starting with 'h' equally: .h, .hpp, .hxx etc.
+# This is set by --headers flag.
+_hpp_headers = set(['h'])
+
 # {str, bool}: a map from error categories to booleans which indicate if the
 # category should be suppressed for every line.
 _global_error_suppressions = {}
 
+def ProcessHppHeadersOption(val):
+  global _hpp_headers
+  try:
+    _hpp_headers = set(val.split(','))
+    # Automatically append to extensions list so it does not have to be set 2 times
+    _valid_extensions.update(_hpp_headers)
+  except ValueError:
+    PrintUsage('Header extensions must be comma separated list.')
+
+def IsHeaderExtension(file_extension):
+  return file_extension in _hpp_headers
 
 def ParseNolintSuppressions(filename, raw_line, linenum, error):
   """Updates the global list of line error-suppressions.
@@ -832,6 +866,7 @@ class _CppLintState(object):
     self._filters_backup = self.filters[:]
     self.counting = 'total'  # In what way are we counting errors?
     self.errors_by_category = {}  # string to int dict storing error counts
+    self.quiet = False  # Suppress non-error messagess?
 
     # output format:
     # "emacs" - format that emacs can parse (default)
@@ -842,6 +877,12 @@ class _CppLintState(object):
     """Sets the output format for errors."""
     self.output_format = output_format
 
+  def SetQuiet(self, quiet):
+    """Sets the module's quiet settings, and returns the previous setting."""
+    last_quiet = self.quiet
+    self.quiet = quiet
+    return last_quiet
+
   def SetVerboseLevel(self, level):
     """Sets the module's verbosity, and returns the previous setting."""
     last_verbose_level = self.verbose_level
@@ -909,7 +950,7 @@ class _CppLintState(object):
     for category, count in self.errors_by_category.iteritems():
       sys.stderr.write('Category \'%s\' errors found: %d\n' %
                        (category, count))
-    sys.stderr.write('Total errors found: %d\n' % self.error_count)
+    sys.stdout.write('Total errors found: %d\n' % self.error_count)
 
 _cpplint_state = _CppLintState()
 
@@ -923,6 +964,14 @@ def _SetOutputFormat(output_format):
   """Sets the module's output format."""
   _cpplint_state.SetOutputFormat(output_format)
 
+def _Quiet():
+  """Return's the module's quiet setting."""
+  return _cpplint_state.quiet
+
+def _SetQuiet(quiet):
+  """Set the module's quiet status, and return previous setting."""
+  return _cpplint_state.SetQuiet(quiet)
+
 
 def _VerboseLevel():
   """Returns the module's verbosity setting."""
@@ -1184,8 +1233,8 @@ def Error(filename, linenum, category, confidence, message):
   if _ShouldPrintError(category, confidence, linenum):
     _cpplint_state.IncrementErrorCount(category)
     if _cpplint_state.output_format == 'vs7':
-      sys.stderr.write('%s(%s):  %s  [%s] [%d]\n' % (
-          filename, linenum, message, category, confidence))
+      sys.stderr.write('%s(%s): error cpplint: [%s] %s [%d]\n' % (
+          filename, linenum, category, message, confidence))
     elif _cpplint_state.output_format == 'eclipse':
       sys.stderr.write('%s:%s: warning: %s  [%s] [%d]\n' % (
           filename, linenum, message, category, confidence))
@@ -1727,6 +1776,30 @@ def GetIndentLevel(line):
   else:
     return 0
 
+def PathSplitToList(path):
+  """Returns the path split into a list by the separator.
+
+  Args:
+    path: An absolute or relative path (e.g. '/a/b/c/' or '../a')
+
+  Returns:
+    A list of path components (e.g. ['a', 'b', 'c]).
+  """
+  lst = []
+  while True:
+    (head, tail) = os.path.split(path)
+    if head == path: # absolute paths end
+      lst.append(head)
+      break
+    if tail == path: # relative paths end
+      lst.append(tail)
+      break
+
+    path = head
+    lst.append(tail)
+
+  lst.reverse()
+  return lst
 
 def GetHeaderGuardCPPVariable(filename):
   """Returns the CPP variable that should be used as a header guard.
@@ -1749,13 +1822,58 @@ def GetHeaderGuardCPPVariable(filename):
 
   fileinfo = FileInfo(filename)
   file_path_from_root = fileinfo.RepositoryName()
-  if _root:
-    suffix = os.sep
-    # On Windows using directory separator will leave us with
-    # "bogus escape error" unless we properly escape regex.
-    if suffix == '\\':
-      suffix += '\\'
-    file_path_from_root = re.sub('^' + _root + suffix, '', file_path_from_root)
+
+  def FixupPathFromRoot():
+    if _root_debug:
+      sys.stderr.write("\n_root fixup, _root = '%s', repository name = '%s'\n"
+          %(_root, fileinfo.RepositoryName()))
+
+    # Process the file path with the --root flag if it was set.
+    if not _root:
+      if _root_debug:
+        sys.stderr.write("_root unspecified\n")
+      return file_path_from_root
+
+    def StripListPrefix(lst, prefix):
+      # f(['x', 'y'], ['w, z']) -> None  (not a valid prefix)
+      if lst[:len(prefix)] != prefix:
+        return None
+      # f(['a, 'b', 'c', 'd'], ['a', 'b']) -> ['c', 'd']
+      return lst[(len(prefix)):]
+
+    # root behavior:
+    #   --root=subdir , lstrips subdir from the header guard
+    maybe_path = StripListPrefix(PathSplitToList(file_path_from_root),
+                                 PathSplitToList(_root))
+
+    if _root_debug:
+      sys.stderr.write("_root lstrip (maybe_path=%s, file_path_from_root=%s," +
+          " _root=%s)\n" %(maybe_path, file_path_from_root, _root))
+
+    if maybe_path:
+      return os.path.join(*maybe_path)
+
+    #   --root=.. , will prepend the outer directory to the header guard
+    full_path = fileinfo.FullName()
+    root_abspath = os.path.abspath(_root)
+
+    maybe_path = StripListPrefix(PathSplitToList(full_path),
+                                 PathSplitToList(root_abspath))
+
+    if _root_debug:
+      sys.stderr.write("_root prepend (maybe_path=%s, full_path=%s, " +
+          "root_abspath=%s)\n" %(maybe_path, full_path, root_abspath))
+
+    if maybe_path:
+      return os.path.join(*maybe_path)
+
+    if _root_debug:
+      sys.stderr.write("_root ignore, returning %s\n" %(file_path_from_root))
+
+    #   --root=FAKE_DIR is ignored
+    return file_path_from_root
+
+  file_path_from_root = FixupPathFromRoot()
   return re.sub(r'[^a-zA-Z0-9]', '_', file_path_from_root).upper() + '_'
 
 
@@ -2752,7 +2870,8 @@ def CheckForNonStandardConstructs(filename, clean_lines, linenum,
   # Look for single-argument constructors that aren't marked explicit.
   # Technically a valid construct, but against style.
   explicit_constructor_match = Match(
-      r'\s+(?:inline\s+)?(explicit\s+)?(?:inline\s+)?%s\s*'
+      r'\s+(?:(?:inline|constexpr)\s+)*(explicit\s+)?'
+      r'(?:(?:inline|constexpr)\s+)*%s\s*'
       r'\(((?:[^()]|\([^()]*\))*)\)'
       % re.escape(base_classname),
       line)
@@ -3044,36 +3163,6 @@ def CheckComment(line, filename, linenum, next_line_start, error):
               'Should have a space between // and comment')
 
 
-def CheckAccess(filename, clean_lines, linenum, nesting_state, error):
-  """Checks for improper use of DISALLOW* macros.
-
-  Args:
-    filename: The name of the current file.
-    clean_lines: A CleansedLines instance containing the file.
-    linenum: The number of the line to check.
-    nesting_state: A NestingState instance which maintains information about
-                   the current stack of nested blocks being parsed.
-    error: The function to call with any errors found.
-  """
-  line = clean_lines.elided[linenum]  # get rid of comments and strings
-
-  matched = Match((r'\s*(DISALLOW_COPY_AND_ASSIGN|'
-                   r'DISALLOW_IMPLICIT_CONSTRUCTORS)'), line)
-  if not matched:
-    return
-  if nesting_state.stack and isinstance(nesting_state.stack[-1], _ClassInfo):
-    if nesting_state.stack[-1].access != 'private':
-      error(filename, linenum, 'readability/constructors', 3,
-            '%s must be in the private: section' % matched.group(1))
-
-  else:
-    # Found DISALLOW* macro outside a class declaration, or perhaps it
-    # was used inside a function when it should have been part of the
-    # class declaration.  We could issue a warning here, but it
-    # probably resulted in a compiler error already.
-    pass
-
-
 def CheckSpacing(filename, clean_lines, linenum, nesting_state, error):
   """Checks for the correctness of various spacing issues in the code.
 
@@ -4271,7 +4360,7 @@ def CheckStyle(filename, clean_lines, linenum, file_extension, nesting_state,
 
   # Check if the line is a header guard.
   is_header_guard = False
-  if file_extension == 'h':
+  if IsHeaderExtension(file_extension):
     cppvar = GetHeaderGuardCPPVariable(filename)
     if (line.startswith('#ifndef %s' % cppvar) or
         line.startswith('#define %s' % cppvar) or
@@ -4310,7 +4399,6 @@ def CheckStyle(filename, clean_lines, linenum, file_extension, nesting_state,
   CheckBraces(filename, clean_lines, linenum, error)
   CheckTrailingSemicolon(filename, clean_lines, linenum, error)
   CheckEmptyBlockBody(filename, clean_lines, linenum, error)
-  CheckAccess(filename, clean_lines, linenum, nesting_state, error)
   CheckSpacing(filename, clean_lines, linenum, nesting_state, error)
   CheckOperatorSpacing(filename, clean_lines, linenum, error)
   CheckParenthesisSpacing(filename, clean_lines, linenum, error)
@@ -4621,7 +4709,7 @@ def CheckLanguage(filename, clean_lines, linenum, file_extension,
   CheckGlobalStatic(filename, clean_lines, linenum, error)
   CheckPrintf(filename, clean_lines, linenum, error)
 
-  if file_extension == 'h':
+  if IsHeaderExtension(file_extension):
     # TODO(unknown): check that 1-arg constructors are explicit.
     #                How to tell it's a constructor?
     #                (handled in CheckForNonStandardConstructs for now)
@@ -4728,7 +4816,7 @@ def CheckLanguage(filename, clean_lines, linenum, file_extension,
   # Check for use of unnamed namespaces in header files.  Registration
   # macros are typically OK, so we allow use of "namespace {" on lines
   # that end with backslashes.
-  if (file_extension == 'h'
+  if (IsHeaderExtension(file_extension)
       and Search(r'\bnamespace\s*{', line)
       and line[-1] != '\\'):
     error(filename, linenum, 'build/namespaces', 4,
@@ -5818,7 +5906,7 @@ def ProcessFileData(filename, file_extension, lines, error,
   RemoveMultiLineComments(filename, lines, error)
   clean_lines = CleansedLines(lines)
 
-  if file_extension == 'h':
+  if IsHeaderExtension(file_extension):
     CheckForHeaderGuard(filename, clean_lines, error)
 
   for line in xrange(clean_lines.NumLines()):
@@ -5887,6 +5975,9 @@ def ProcessConfigOverrides(filename):
             if base_name:
               pattern = re.compile(val)
               if pattern.match(base_name):
+                if _cpplint_state.quiet:
+                  # Suppress "Ignoring file" warning when using --quiet.
+                  return False
                 sys.stderr.write('Ignoring "%s": file excluded by "%s". '
                                  'File path component "%s" matches '
                                  'pattern "%s"\n' %
@@ -5900,7 +5991,10 @@ def ProcessConfigOverrides(filename):
                 sys.stderr.write('Line length must be numeric.')
           elif name == 'root':
             global _root
-            _root = val
+            # root directories are specified relative to CPPLINT.cfg dir.
+            _root = os.path.join(os.path.dirname(cfg_file), val)
+          elif name == 'headers':
+            ProcessHppHeadersOption(val)
           else:
             sys.stderr.write(
                 'Invalid configuration option (%s) in file %s\n' %
@@ -5935,6 +6029,7 @@ def ProcessFile(filename, vlevel, extra_check_functions=[]):
 
   _SetVerboseLevel(vlevel)
   _BackupFilters()
+  old_errors = _cpplint_state.error_count
 
   if not ProcessConfigOverrides(filename):
     _RestoreFilters()
@@ -6003,7 +6098,10 @@ def ProcessFile(filename, vlevel, extra_check_functions=[]):
         Error(filename, linenum, 'whitespace/newline', 1,
               'Unexpected \\r (^M) found; better to use only \\n')
 
-  sys.stderr.write('Done processing %s\n' % filename)
+  # Suppress printing anything if --quiet was passed unless the error
+  # count has increased after processing this file.
+  if not _cpplint_state.quiet or old_errors != _cpplint_state.error_count:
+    sys.stdout.write('Done processing %s\n' % filename)
   _RestoreFilters()
 
 
@@ -6046,13 +6144,16 @@ def ParseArguments(args):
                                                  'filter=',
                                                  'root=',
                                                  'linelength=',
-                                                 'extensions='])
+                                                 'extensions=',
+                                                 'headers=',
+                                                 'quiet'])
   except getopt.GetoptError:
     PrintUsage('Invalid arguments.')
 
   verbosity = _VerboseLevel()
   output_format = _OutputFormat()
   filters = ''
+  quiet = _Quiet()
   counting_style = ''
 
   for (opt, val) in opts:
@@ -6062,6 +6163,8 @@ def ParseArguments(args):
       if val not in ('emacs', 'vs7', 'eclipse'):
         PrintUsage('The only allowed output formats are emacs, vs7 and eclipse.')
       output_format = val
+    elif opt == '--quiet':
+      quiet = True
     elif opt == '--verbose':
       verbosity = int(val)
     elif opt == '--filter':
@@ -6087,11 +6190,14 @@ def ParseArguments(args):
           _valid_extensions = set(val.split(','))
       except ValueError:
           PrintUsage('Extensions must be comma seperated list.')
+    elif opt == '--headers':
+      ProcessHppHeadersOption(val)
 
   if not filenames:
     PrintUsage('No files were specified.')
 
   _SetOutputFormat(output_format)
+  _SetQuiet(quiet)
   _SetVerboseLevel(verbosity)
   _SetFilters(filters)
   _SetCountingStyle(counting_style)
@@ -6112,7 +6218,9 @@ def main():
   _cpplint_state.ResetErrorCounts()
   for filename in filenames:
     ProcessFile(filename, _cpplint_state.verbose_level)
-  _cpplint_state.PrintErrorCounts()
+  # If --quiet is passed, suppress printing error count unless there are errors.
+  if not _cpplint_state.quiet or _cpplint_state.error_count > 0:
+    _cpplint_state.PrintErrorCounts()
 
   sys.exit(_cpplint_state.error_count > 0)
 
diff --git a/tools/cpplint.py-update b/tools/cpplint.py-update
index 1767e41..4af4389 100755
--- a/tools/cpplint.py-update
+++ b/tools/cpplint.py-update
@@ -47,6 +47,7 @@ main() {
   # Download cpplint.py from upstream.
   local cpplint_py="${SCRIPT_DIR}/cpplint.py"
   wget "${GITHUB_URL}/cpplint/cpplint.py" -O "${cpplint_py}"
+  sed -i '2i# pylint: skip-file' "${cpplint_py}"
   chmod +x "${cpplint_py}"
 }
 
diff --git a/tools/google-java-format.py b/tools/google-java-format.py
index fcc5dd5..7c80124 100755
--- a/tools/google-java-format.py
+++ b/tools/google-java-format.py
@@ -29,6 +29,9 @@ if sys.path[0] != _path:
     sys.path.insert(0, _path)
 del _path
 
+# We have to import our local modules after the sys.path tweak.  We can't use
+# relative imports because this is an executable program, not a module.
+# pylint: disable=wrong-import-position
 import rh.shell
 import rh.utils
 
@@ -55,6 +58,9 @@ def get_parser():
     # style.
     parser.add_argument('--noaosp', action='store_true',
                         help='Use google-java-format style instead of AOSP style.')
+    parser.add_argument('files', nargs='*',
+                        help='If specified, only consider differences in '
+                             'these files.')
     return parser
 
 
@@ -111,7 +117,8 @@ def main(argv):
 
     # TODO: Delegate to the tool once this issue is resolved:
     # https://github.com/google/google-java-format/issues/107
-    diff_cmd = ['git', 'diff', '-U0', '%s^!' % opts.commit]
+    diff_cmd = ['git', 'diff', '--no-ext-diff', '-U0', '%s^!' % opts.commit]
+    diff_cmd.extend(['--'] + opts.files)
     diff = rh.utils.run_command(diff_cmd, capture_output=True).output
 
     cmd = [opts.google_java_format_diff, '-p1']
@@ -130,7 +137,7 @@ def main(argv):
                                   input=diff,
                                   capture_output=True,
                                   extra_env=extra_env).output
-    if stdout != '':
+    if stdout:
         print('One or more files in your commit have Java formatting errors.')
         if platform.system() == 'Windows':
             print('You can run `python %s --fix %s` to fix this' %
diff --git a/tools/patched/ElementPath.py b/tools/patched/ElementPath.py
index 4a626d7..52e65f0 100644
--- a/tools/patched/ElementPath.py
+++ b/tools/patched/ElementPath.py
@@ -255,7 +255,7 @@ def iterfind(elem, path, namespaces=None):
             _cache.clear()
         if path[:1] == "/":
             raise SyntaxError("cannot use absolute path on element")
-        next = iter(xpath_tokenizer(path, namespaces)).next
+        next = iter(xpath_tokenizer(path, namespaces)).__next__
         token = next()
         selector = []
         while 1:
@@ -282,7 +282,7 @@ def iterfind(elem, path, namespaces=None):
 
 def find(elem, path, namespaces=None):
     try:
-        return iterfind(elem, path, namespaces).next()
+        return next(iterfind(elem, path, namespaces))
     except StopIteration:
         return None
 
@@ -297,7 +297,7 @@ def findall(elem, path, namespaces=None):
 
 def findtext(elem, path, default=None, namespaces=None):
     try:
-        elem = iterfind(elem, path, namespaces).next()
+        elem = next(iterfind(elem, path, namespaces))
         return elem.text or ""
     except StopIteration:
         return default
diff --git a/tools/patched/ElementTree.py b/tools/patched/ElementTree.py
index 18a1173..b2cb5f0 100644
--- a/tools/patched/ElementTree.py
+++ b/tools/patched/ElementTree.py
@@ -217,7 +217,7 @@ class Element(object):
     # constructor
 
     def __init__(self, tag, attrib={},
-                 attrib_order=defaultdict(lambda: sys.maxint, {}),
+                 attrib_order=defaultdict(lambda: sys.maxsize, {}),
                  **extra):
         attrib = attrib.copy()
         attrib.update(extra)
@@ -263,7 +263,7 @@ class Element(object):
     def __len__(self):
         return len(self._children)
 
-    def __nonzero__(self):
+    def __bool__(self):
         warnings.warn(
             "The behavior of this method will change in future versions.  "
             "Use specific 'len(elem)' or 'elem is not None' test instead.",
@@ -459,7 +459,7 @@ class Element(object):
     # @defreturn list of strings
 
     def keys(self):
-        return self.attrib.keys()
+        return list(self.attrib.keys())
 
     ##
     # Gets element attributes, as a sequence.  The attributes are
@@ -469,7 +469,7 @@ class Element(object):
     # @defreturn list of (string, string) tuples
 
     def items(self):
-        return self.attrib.items()
+        return list(self.attrib.items())
 
     ##
     # Creates a tree iterator.  The iterator loops over this element
@@ -513,7 +513,7 @@ class Element(object):
 
     def itertext(self):
         tag = self.tag
-        if not isinstance(tag, basestring) and tag is not None:
+        if not isinstance(tag, str) and tag is not None:
             return
         if self.text:
             yield self.text
@@ -895,12 +895,12 @@ def _namespaces(elem, encoding, default_namespace=None):
         if isinstance(tag, QName):
             if tag.text not in qnames:
                 add_qname(tag.text)
-        elif isinstance(tag, basestring):
+        elif isinstance(tag, str):
             if tag not in qnames:
                 add_qname(tag)
         elif tag is not None and tag is not Comment and tag is not PI:
             _raise_serialization_error(tag)
-        for key, value in elem.items():
+        for key, value in list(elem.items()):
             if isinstance(key, QName):
                 key = key.text
             if key not in qnames:
@@ -928,10 +928,10 @@ def _serialize_xml(write, elem, encoding, qnames, namespaces):
                 _serialize_xml(write, e, encoding, qnames, None)
         else:
             write("<" + tag)
-            items = elem.items()
+            items = list(elem.items())
             if items or namespaces:
                 if namespaces:
-                    for v, k in sorted(namespaces.items(),
+                    for v, k in sorted(list(namespaces.items()),
                                        key=lambda x: x[1]):  # sort on prefix
                         if k:
                             k = ":" + k
@@ -984,10 +984,10 @@ def _serialize_html(write, elem, encoding, qnames, namespaces):
                 _serialize_html(write, e, encoding, qnames, None)
         else:
             write("<" + tag)
-            items = elem.items()
+            items = list(elem.items())
             if items or namespaces:
                 if namespaces:
-                    for v, k in sorted(namespaces.items(),
+                    for v, k in sorted(list(namespaces.items()),
                                        key=lambda x: x[1]):  # sort on prefix
                         if k:
                             k = ":" + k
@@ -1046,7 +1046,7 @@ _serialize = {
 def register_namespace(prefix, uri):
     if re.match("ns\d+$", prefix):
         raise ValueError("Prefix format reserved for internal use")
-    for k, v in _namespace_map.items():
+    for k, v in list(_namespace_map.items()):
         if k == uri or v == prefix:
             del _namespace_map[k]
     _namespace_map[uri] = prefix
@@ -1268,7 +1268,7 @@ class _IterParseIterator(object):
             else:
                 raise ValueError("unknown event %r" % event)
 
-    def next(self):
+    def __next__(self):
         while 1:
             try:
                 item = self._events[self._index]
@@ -1565,7 +1565,7 @@ class XMLParser(object):
         fixtext = self._fixtext
         tag = fixname(tag)
         attrib = {}
-        for key, value in attrib_in.items():
+        for key, value in list(attrib_in.items()):
             attrib[fixname(key)] = fixtext(value)
         return self.target.start(tag, attrib)
 
@@ -1575,7 +1575,7 @@ class XMLParser(object):
         tag = fixname(tag)
         attrib = {}
         # noinspection PyArgumentList
-        attrib_order = defaultdict(lambda: sys.maxint, {})
+        attrib_order = defaultdict(lambda: sys.maxsize, {})
         if attrib_in:
             for i in range(0, len(attrib_in), 2):
                 name = fixname(attrib_in[i])
@@ -1679,7 +1679,7 @@ class XMLParser(object):
     def feed(self, data):
         try:
             self._parser.Parse(data, 0)
-        except self._error, v:
+        except self._error as v:
             self._raiseerror(v)
 
     ##
@@ -1691,7 +1691,7 @@ class XMLParser(object):
     def close(self):
         try:
             self._parser.Parse("", 1) # end of data
-        except self._error, v:
+        except self._error as v:
             self._raiseerror(v)
         tree = self.target.close()
         del self.target, self._parser # get rid of circular references
diff --git a/tools/pylint.py b/tools/pylint.py
index 45afe64..7885dcf 100755
--- a/tools/pylint.py
+++ b/tools/pylint.py
@@ -19,12 +19,15 @@
 from __future__ import print_function
 
 import argparse
+import errno
 import os
 import sys
 
+
 DEFAULT_PYLINTRC_PATH = os.path.join(
     os.path.dirname(os.path.realpath(__file__)), 'pylintrc')
 
+
 def get_parser():
     """Return a command line parser."""
     parser = argparse.ArgumentParser(description=__doc__)
@@ -45,12 +48,16 @@ def main(argv):
     parser = get_parser()
     opts, unknown = parser.parse_known_args(argv)
 
-    pylintrc = DEFAULT_PYLINTRC_PATH if not opts.no_rcfile else None
-
     cmd = [opts.executable_path]
-    if pylintrc:
-        # If we pass a non-existent rcfile to pylint, it'll happily ignore it.
-        assert os.path.exists(pylintrc), 'Could not find %s' % pylintrc
+    if not opts.no_rcfile:
+        # We assume pylint is running in the top directory of the project,
+        # so load the pylintrc file from there if it's available.
+        pylintrc = os.path.abspath('pylintrc')
+        if not os.path.exists(pylintrc):
+            pylintrc = DEFAULT_PYLINTRC_PATH
+            # If we pass a non-existent rcfile to pylint, it'll happily ignore
+            # it.
+            assert os.path.exists(pylintrc), 'Could not find %s' % pylintrc
         cmd += ['--rcfile', pylintrc]
 
     cmd += unknown + opts.files
@@ -58,7 +65,18 @@ def main(argv):
     if opts.init_hook:
         cmd += ['--init-hook', opts.init_hook]
 
-    os.execvp(cmd[0], cmd)
+    try:
+        os.execvp(cmd[0], cmd)
+        return 0
+    except OSError as e:
+        if e.errno == errno.ENOENT:
+            print('%s: unable to run `%s`: %s' % (__file__, cmd[0], e),
+                  file=sys.stderr)
+            print('%s: Try installing pylint: sudo apt-get install %s' %
+                  (__file__, os.path.basename(cmd[0])), file=sys.stderr)
+            return 1
+
+        raise
 
 
 if __name__ == '__main__':
diff --git a/tools/pylintrc b/tools/pylintrc
index 599bb88..68c74ef 100644
--- a/tools/pylintrc
+++ b/tools/pylintrc
@@ -34,9 +34,16 @@ persistent=yes
 # List of plugins (as comma separated values of python modules names) to load,
 # usually to register additional checkers.
 load-plugins=
+    pylint.extensions.bad_builtin,
+    pylint.extensions.check_elif,
+    pylint.extensions.docstyle,
+    pylint.extensions.emptystring,
+    pylint.extensions.overlapping_exceptions,
+    pylint.extensions.redefined_variable_type,
 
-# Use multiple processes to speed up Pylint.
-jobs=1
+# Use multiple processes to speed up Pylint.  A value of 0 autodetects available
+# processors.
+jobs=0
 
 # Allow loading of arbitrary C extensions. Extensions are imported into the
 # active Python interpreter and may run arbitrary code.
@@ -65,7 +72,70 @@ confidence=
 # Enable the message, report, category or checker with the given id(s). You can
 # either give multiple identifier separated by comma (,) or put this option
 # multiple time. See also the "--disable" option for examples.
-#enable=
+enable=
+    apply-builtin,
+    backtick,
+    bad-python3-import,
+    basestring-builtin,
+    buffer-builtin,
+    cmp-builtin,
+    cmp-method,
+    coerce-builtin,
+    coerce-method,
+    delslice-method,
+    deprecated-itertools-function,
+    deprecated-str-translate-call,
+    deprecated-string-function,
+    deprecated-types-field,
+    dict-items-not-iterating,
+    dict-iter-method,
+    dict-keys-not-iterating,
+    dict-values-not-iterating,
+    dict-view-method,
+    div-method,
+    exception-message-attribute,
+    execfile-builtin,
+    file-builtin,
+    filter-builtin-not-iterating,
+    getslice-method,
+    hex-method,
+    idiv-method,
+    import-star-module-level,
+    indexing-exception,
+    input-builtin,
+    intern-builtin,
+    invalid-str-codec,
+    long-builtin,
+    long-suffix,
+    map-builtin-not-iterating,
+    metaclass-assignment,
+    next-method-called,
+    next-method-defined,
+    nonzero-method,
+    oct-method,
+    old-division,
+    old-ne-operator,
+    old-octal-literal,
+    old-raise-syntax,
+    parameter-unpacking,
+    print-statement,
+    raising-string,
+    range-builtin-not-iterating,
+    raw_input-builtin,
+    rdiv-method,
+    reduce-builtin,
+    reload-builtin,
+    round-builtin,
+    setslice-method,
+    standarderror-builtin,
+    sys-max-int,
+    unichr-builtin,
+    unicode-builtin,
+    unpacking-in-except,
+    using-cmp-argument,
+    xrange-builtin,
+    zip-builtin-not-iterating,
+
 
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
@@ -77,8 +147,10 @@ confidence=
 # no Warning level messages displayed, use"--disable=all --enable=classes
 # --disable=W"
 # We leave many of the style warnings to judgement/peer review.
+# useless-object-inheritance: We disable this for Python 2 compatibility.
 disable=
     fixme,
+    file-ignored,
     invalid-name,
     locally-disabled,
     locally-enabled,
@@ -94,6 +166,7 @@ disable=
     too-many-public-methods,
     too-many-return-statements,
     too-many-statements,
+    useless-object-inheritance,
 
 
 [REPORTS]
@@ -111,6 +184,9 @@ files-output=no
 # Tells whether to display a full report or only the messages
 reports=no
 
+# Activate the evaluation score.
+score=no
+
 # Python expression which should return a note less than 10 (10 is the highest
 # note). You have access to the variables errors warning, statement which
 # respectively contain the number of errors / warnings messages and the total
diff --git a/tools/spelling.txt b/tools/spelling.txt
index 946caf3..9a058cf 100644
--- a/tools/spelling.txt
+++ b/tools/spelling.txt
@@ -16,6 +16,7 @@ absense||absence
 absolut||absolute
 absoulte||absolute
 acccess||access
+acceess||access
 acceleratoin||acceleration
 accelleration||acceleration
 accesing||accessing
@@ -39,17 +40,22 @@ achitecture||architecture
 acient||ancient
 acitions||actions
 acitve||active
-acknowldegement||acknowldegement
+acknowldegement||acknowledgment
 acknowledgement||acknowledgment
 ackowledge||acknowledge
 ackowledged||acknowledged
 acording||according
 activete||activate
+actived||activated
+actualy||actually
 acumulating||accumulating
+acumulator||accumulator
 adapater||adapter
 addional||additional
 additionaly||additionally
+additonal||additional
 addres||address
+adddress||address
 addreses||addresses
 addresss||address
 aditional||additional
@@ -60,16 +66,26 @@ adress||address
 adresses||addresses
 adviced||advised
 afecting||affecting
+againt||against
 agaist||against
+aggreataon||aggregation
+aggreation||aggregation
 albumns||albums
 alegorical||allegorical
+algined||aligned
 algorith||algorithm
 algorithmical||algorithmically
 algoritm||algorithm
 algoritms||algorithms
 algorrithm||algorithm
 algorritm||algorithm
+aligment||alignment
+alignement||alignment
 allign||align
+alligned||aligned
+alllocate||allocate
+alloated||allocated
+allocatote||allocate
 allocatrd||allocated
 allocte||allocate
 allpication||application
@@ -84,6 +100,10 @@ alue||value
 ambigious||ambiguous
 amoung||among
 amout||amount
+an union||a union
+an user||a user
+an userspace||a userspace
+an one||a one
 analysator||analyzer
 ang||and
 anniversery||anniversary
@@ -96,6 +116,7 @@ appearence||appearance
 applicaion||application
 appliction||application
 applictions||applications
+applys||applies
 appplications||applications
 appropiate||appropriate
 appropriatly||appropriately
@@ -115,6 +136,7 @@ arraival||arrival
 artifical||artificial
 artillary||artillery
 asign||assign
+asser||assert
 assertation||assertion
 assiged||assigned
 assigment||assignment
@@ -130,9 +152,11 @@ asycronous||asynchronous
 asynchnous||asynchronous
 atomatically||automatically
 atomicly||atomically
+atempt||attempt
 attachement||attachment
 attched||attached
 attemps||attempts
+attemping||attempting
 attruibutes||attributes
 authentification||authentication
 automaticaly||automatically
@@ -152,6 +176,7 @@ availale||available
 availavility||availability
 availble||available
 availiable||available
+availible||available
 avalable||available
 avaliable||available
 aysnc||async
@@ -163,6 +188,7 @@ bakup||backup
 baloon||balloon
 baloons||balloons
 bandwith||bandwidth
+banlance||balance
 batery||battery
 beacuse||because
 becasue||because
@@ -183,13 +209,18 @@ broadcat||broadcast
 cacluated||calculated
 caculation||calculation
 calender||calendar
+calescing||coalescing
 calle||called
+callibration||calibration
 calucate||calculate
 calulate||calculate
 cancelation||cancellation
+cancle||cancel
 capabilites||capabilities
+capabilty||capability
 capabitilies||capabilities
 capatibilities||capabilities
+capapbilities||capabilities
 carefuly||carefully
 cariage||carriage
 catagory||category
@@ -198,6 +229,7 @@ challange||challenge
 challanges||challenges
 chanell||channel
 changable||changeable
+chanined||chained
 channle||channel
 channnel||channel
 charachter||character
@@ -223,6 +255,7 @@ claread||cleared
 clared||cleared
 closeing||closing
 clustred||clustered
+coexistance||coexistence
 collapsable||collapsible
 colorfull||colorful
 comand||command
@@ -234,6 +267,9 @@ commited||committed
 commiting||committing
 committ||commit
 commoditiy||commodity
+comsume||consume
+comsumer||consumer
+comsuming||consuming
 compability||compatibility
 compaibility||compatibility
 compatability||compatibility
@@ -244,9 +280,11 @@ compatiblity||compatibility
 competion||completion
 compilant||compliant
 compleatly||completely
+completition||completion
 completly||completely
 complient||compliant
 componnents||components
+compoment||component
 compres||compress
 compresion||compression
 comression||compression
@@ -254,9 +292,12 @@ comunication||communication
 conbination||combination
 conditionaly||conditionally
 conected||connected
+connecetd||connected
+configuartion||configuration
 configuratoin||configuration
 configuraton||configuration
 configuretion||configuration
+configutation||configuration
 conider||consider
 conjuction||conjunction
 connectinos||connections
@@ -269,15 +310,19 @@ containts||contains
 contaisn||contains
 contant||contact
 contence||contents
+continious||continuous
 continous||continuous
 continously||continuously
 continueing||continuing
 contraints||constraints
+contol||control
+contoller||controller
 controled||controlled
 controler||controller
 controll||control
 contruction||construction
 contry||country
+conuntry||country
 convertion||conversion
 convertor||converter
 convienient||convenient
@@ -287,16 +332,19 @@ correponding||corresponding
 correponds||corresponds
 correspoding||corresponding
 cotrol||control
+cound||could
 couter||counter
 coutner||counter
 cryptocraphic||cryptographic
 cunter||counter
 curently||currently
+cylic||cyclic
 dafault||default
 deafult||default
 deamon||daemon
 decompres||decompress
 decription||description
+dectected||detected
 defailt||default
 defferred||deferred
 definate||definite
@@ -305,18 +353,24 @@ defintion||definition
 defintions||definitions
 defualt||default
 defult||default
+deintializing||deinitializing
+deintialize||deinitialize
+deintialized||deinitialized
 deivce||device
 delared||declared
 delare||declare
 delares||declares
 delaring||declaring
 delemiter||delimiter
+demodualtor||demodulator
+demension||dimension
 dependancies||dependencies
 dependancy||dependency
 dependant||dependent
 depreacted||deprecated
 depreacte||deprecate
 desactivate||deactivate
+desciptor||descriptor
 desciptors||descriptors
 descripton||description
 descrition||description
@@ -324,11 +378,13 @@ descritptor||descriptor
 desctiptor||descriptor
 desriptor||descriptor
 desriptors||descriptors
+destionation||destination
 destory||destroy
 destoryed||destroyed
 destorys||destroys
 destroied||destroyed
 detabase||database
+deteced||detected
 develope||develop
 developement||development
 developped||developed
@@ -343,12 +399,18 @@ dictionnary||dictionary
 didnt||didn't
 diferent||different
 differrence||difference
+diffrent||different
+diffrentiate||differentiate
 difinition||definition
+dimesions||dimensions
 diplay||display
 direectly||directly
+disassocation||disassociation
 disapear||disappear
 disapeared||disappeared
 disappared||disappeared
+disble||disable
+disbled||disabled
 disconnet||disconnect
 discontinous||discontinuous
 dispertion||dispersion
@@ -369,18 +431,23 @@ easilly||easily
 ecspecially||especially
 edditable||editable
 editting||editing
+efective||effective
 efficently||efficiently
 ehther||ether
 eigth||eight
+elementry||elementary
 eletronic||electronic
+embeded||embedded
 enabledi||enabled
 enchanced||enhanced
 encorporating||incorporating
 encrupted||encrypted
 encrypiton||encryption
+encryptio||encryption
 endianess||endianness
 enhaced||enhanced
 enlightnment||enlightenment
+entrys||entries
 enocded||encoded
 enterily||entirely
 enviroiment||environment
@@ -392,12 +459,14 @@ equiped||equipped
 equivelant||equivalent
 equivilant||equivalent
 eror||error
+errorr||error
 estbalishment||establishment
 etsablishment||establishment
 etsbalishment||establishment
 excecutable||executable
 exceded||exceeded
 excellant||excellent
+exeed||exceed
 existance||existence
 existant||existent
 exixt||exist
@@ -408,6 +477,7 @@ expecially||especially
 explicite||explicit
 explicitely||explicitly
 explict||explicit
+explictely||explicitly
 explictly||explicitly
 expresion||expression
 exprimental||experimental
@@ -415,12 +485,18 @@ extened||extended
 extensability||extensibility
 extention||extension
 extracter||extractor
+falied||failed
 faild||failed
 faill||fail
+failied||failed
+faillure||failure
 failue||failure
 failuer||failure
+failng||failing
 faireness||fairness
+falied||failed
 faliure||failure
+fallbck||fallback
 familar||familiar
 fatser||faster
 feauture||feature
@@ -428,6 +504,8 @@ feautures||features
 fetaure||feature
 fetaures||features
 fileystem||filesystem
+fimware||firmware
+firware||firmware
 finanize||finalize
 findn||find
 finilizes||finalizes
@@ -435,11 +513,14 @@ finsih||finish
 flusing||flushing
 folloing||following
 followign||following
+followings||following
 follwing||following
+fonud||found
 forseeable||foreseeable
 forse||force
 fortan||fortran
 forwardig||forwarding
+framming||framing
 framwork||framework
 frequncy||frequency
 frome||from
@@ -458,12 +539,14 @@ futhermore||furthermore
 futrue||future
 gaurenteed||guaranteed
 generiously||generously
+genereate||generate
 genric||generic
 globel||global
 grabing||grabbing
 grahical||graphical
 grahpical||graphical
 grapic||graphic
+grranted||granted
 guage||gauge
 guarenteed||guaranteed
 guarentee||guarantee
@@ -475,12 +558,16 @@ happend||happened
 harware||hardware
 heirarchically||hierarchically
 helpfull||helpful
+hybernate||hibernate
 hierachy||hierarchy
 hierarchie||hierarchy
 howver||however
 hsould||should
+hypervior||hypervisor
 hypter||hyper
 identidier||identifier
+iligal||illegal
+illigal||illegal
 imblance||imbalance
 immeadiately||immediately
 immedaite||immediate
@@ -494,16 +581,19 @@ implemenation||implementation
 implementaiton||implementation
 implementated||implemented
 implemention||implementation
+implementd||implemented
 implemetation||implementation
 implemntation||implementation
 implentation||implementation
 implmentation||implementation
 implmenting||implementing
+incative||inactive
 incomming||incoming
 incompatabilities||incompatibilities
 incompatable||incompatible
 inconsistant||inconsistent
 increas||increase
+incremeted||incremented
 incrment||increment
 indendation||indentation
 indended||intended
@@ -511,6 +601,7 @@ independant||independent
 independantly||independently
 independed||independent
 indiate||indicate
+indicat||indicate
 inexpect||inexpected
 infomation||information
 informatiom||information
@@ -519,11 +610,13 @@ informtion||information
 infromation||information
 ingore||ignore
 inital||initial
+initalized||initialized
 initalised||initialized
 initalise||initialize
 initalize||initialize
 initation||initiation
 initators||initiators
+initialiazation||initialization
 initializiation||initialization
 initialzed||initialized
 initilization||initialization
@@ -531,6 +624,7 @@ initilize||initialize
 inofficial||unofficial
 insititute||institute
 instal||install
+instanciated||instantiated
 inteface||interface
 integreated||integrated
 integrety||integrity
@@ -544,6 +638,7 @@ interger||integer
 intermittant||intermittent
 internel||internal
 interoprability||interoperability
+interuupt||interrupt
 interrface||interface
 interrrupt||interrupt
 interrup||interrupt
@@ -552,20 +647,30 @@ interruptted||interrupted
 interupted||interrupted
 interupt||interrupt
 intial||initial
+intialisation||initialisation
+intialised||initialised
+intialise||initialise
+intialization||initialization
 intialized||initialized
 intialize||initialize
 intregral||integral
 intrrupt||interrupt
+intterrupt||interrupt
 intuative||intuitive
 invaid||invalid
-invalde||invald
+invald||invalid
+invalde||invalid
 invalide||invalid
+invalidiate||invalidate
+invalud||invalid
 invididual||individual
 invokation||invocation
 invokations||invocations
 irrelevent||irrelevant
 isnt||isn't
 isssue||issue
+iternations||iterations
+itertation||iteration
 itslef||itself
 jave||java
 jeffies||jiffies
@@ -618,18 +723,27 @@ messags||messages
 messgaes||messages
 messsage||message
 messsages||messages
+micropone||microphone
 microprocesspr||microprocessor
 milliseonds||milliseconds
+minium||minimum
+minimam||minimum
 minumum||minimum
+misalinged||misaligned
 miscelleneous||miscellaneous
 misformed||malformed
 mispelled||misspelled
 mispelt||misspelt
+mising||missing
+mismactch||mismatch
+missmanaged||mismanaged
+missmatch||mismatch
 miximum||maximum
 mmnemonic||mnemonic
 mnay||many
-modeled||modelled
 modulues||modules
+momery||memory
+memomry||memory
 monochorome||monochrome
 monochromo||monochrome
 monocrome||monochrome
@@ -640,6 +754,7 @@ multidimensionnal||multidimensional
 multple||multiple
 mumber||number
 muticast||multicast
+mutilcast||multicast
 mutiple||multiple
 mutli||multi
 nams||names
@@ -649,6 +764,7 @@ neccecary||necessary
 neccesary||necessary
 neccessary||necessary
 necesary||necessary
+neded||needed
 negaive||negative
 negoitation||negotiation
 negotation||negotiation
@@ -668,8 +784,11 @@ occurances||occurrences
 occured||occurred
 occurence||occurrence
 occure||occurred
+occured||occurred
 occuring||occurring
 offet||offset
+omited||omitted
+omiting||omitting
 omitt||omit
 ommiting||omitting
 ommitted||omitted
@@ -681,13 +800,19 @@ optionnal||optional
 optmizations||optimizations
 orientatied||orientated
 orientied||oriented
+orignal||original
 otherise||otherwise
 ouput||output
+oustanding||outstanding
 overaall||overall
 overhread||overhead
 overlaping||overlapping
+overide||override
+overrided||overridden
 overriden||overridden
 overun||overrun
+overwritting||overwriting
+overwriten||overwritten
 pacakge||package
 pachage||package
 packacge||package
@@ -698,6 +823,7 @@ pakage||package
 pallette||palette
 paln||plan
 paramameters||parameters
+paramaters||parameters
 paramater||parameter
 parametes||parameters
 parametised||parametrised
@@ -705,6 +831,7 @@ paramter||parameter
 paramters||parameters
 particuarly||particularly
 particularily||particularly
+partiton||partition
 pased||passed
 passin||passing
 pathes||paths
@@ -718,17 +845,21 @@ permissons||permissions
 peroid||period
 persistance||persistence
 persistant||persistent
+plalform||platform
 platfrom||platform
 plattform||platform
 pleaes||please
 ploting||plotting
 plugable||pluggable
 poinnter||pointer
+pointeur||pointer
 poiter||pointer
 posible||possible
 positon||position
 possibilites||possibilities
 powerfull||powerful
+preample||preamble
+preapre||prepare
 preceeded||preceded
 preceeding||preceding
 preceed||precede
@@ -752,6 +883,7 @@ procceed||proceed
 proccesors||processors
 procesed||processed
 proces||process
+procesing||processing
 processessing||processing
 processess||processes
 processpr||processor
@@ -780,11 +912,13 @@ protable||portable
 protcol||protocol
 protecion||protection
 protocoll||protocol
+promixity||proximity
 psudo||pseudo
 psuedo||pseudo
 psychadelic||psychedelic
 pwoer||power
 quering||querying
+randomally||randomly
 raoming||roaming
 reasearcher||researcher
 reasearchers||researchers
@@ -801,6 +935,7 @@ recommanded||recommended
 recyle||recycle
 redircet||redirect
 redirectrion||redirection
+reename||rename
 refcounf||refcount
 refence||reference
 refered||referred
@@ -811,8 +946,10 @@ refernnce||reference
 refrence||reference
 registerd||registered
 registeresd||registered
+registerred||registered
 registes||registers
 registraration||registration
+regsiter||register
 regster||register
 regualar||regular
 reguator||regulator
@@ -830,6 +967,7 @@ replys||replies
 reponse||response
 representaion||representation
 reqeust||request
+requestied||requested
 requiere||require
 requirment||requirement
 requred||required
@@ -837,6 +975,7 @@ requried||required
 requst||request
 reseting||resetting
 resizeable||resizable
+resouce||resource
 resouces||resources
 resoures||resources
 responce||response
@@ -852,6 +991,7 @@ reudce||reduce
 reuest||request
 reuqest||request
 reutnred||returned
+revsion||revision
 rmeoved||removed
 rmeove||remove
 rmeoves||removes
@@ -923,6 +1063,7 @@ spinlcok||spinlock
 spinock||spinlock
 splitted||split
 spreaded||spread
+spurrious||spurious
 sructure||structure
 stablilization||stabilization
 staically||statically
@@ -937,14 +1078,18 @@ straming||streaming
 struc||struct
 structres||structures
 stuct||struct
+strucuture||structure
 stucture||structure
 sturcture||structure
 subdirectoires||subdirectories
 suble||subtle
 substract||subtract
+submition||submission
 succesfully||successfully
 succesful||successful
+successed||succeeded
 successfull||successful
+successfuly||successfully
 sucessfully||successfully
 sucess||success
 superflous||superfluous
@@ -952,21 +1097,33 @@ superseeded||superseded
 suplied||supplied
 suported||supported
 suport||support
+supportet||supported
 suppored||supported
 supportin||supporting
 suppoted||supported
 suppported||supported
 suppport||support
 supress||suppress
+surpressed||suppressed
 surpresses||suppresses
 susbsystem||subsystem
+suspeneded||suspended
 suspicously||suspiciously
 swaping||swapping
 switchs||switches
+swith||switch
+swithable||switchable
+swithc||switch
+swithced||switched
+swithcing||switching
+swithed||switched
+swithing||switching
+swtich||switch
 symetric||symmetric
 synax||syntax
 synchonized||synchronized
 syncronize||synchronize
+syncronized||synchronized
 syncronizing||synchronizing
 syncronus||synchronous
 syste||system
@@ -978,49 +1135,71 @@ targetting||targeting
 teh||the
 temorary||temporary
 temproarily||temporarily
+therfore||therefore
 thier||their
 threds||threads
 threshhold||threshold
+thresold||threshold
 throught||through
+troughput||throughput
 thses||these
 tiggered||triggered
 tipically||typically
+timout||timeout
 tmis||this
 torerable||tolerable
 tramsmitted||transmitted
 tramsmit||transmit
+tranasction||transaction
 tranfer||transfer
 transciever||transceiver
-transferd||transferrd
+transferd||transferred
 transfered||transferred
 transfering||transferring
 transision||transition
 transmittd||transmitted
 transormed||transformed
+trasfer||transfer
 trasmission||transmission
 treshold||threshold
 trigerring||triggering
 trun||turn
+tunning||tuning
 ture||true
 tyep||type
 udpate||update
 uesd||used
+uncommited||uncommitted
 unconditionaly||unconditionally
 underun||underrun
 unecessary||unnecessary
 unexecpted||unexpected
+unexepected||unexpected
+unexpcted||unexpected
 unexpectd||unexpected
 unexpeted||unexpected
+unexpexted||unexpected
 unfortunatelly||unfortunately
 unifiy||unify
 unintialized||uninitialized
+unkmown||unknown
 unknonw||unknown
 unknow||unknown
 unkown||unknown
+unneded||unneeded
+unneccecary||unnecessary
+unneccesary||unnecessary
+unneccessary||unnecessary
+unnecesary||unnecessary
 unneedingly||unnecessarily
+unnsupported||unsupported
+unmached||unmatched
+unregester||unregister
 unresgister||unregister
+unrgesiter||unregister
 unsinged||unsigned
 unstabel||unstable
+unsolicitied||unsolicited
 unsuccessfull||unsuccessful
 unsuported||unsupported
 untill||until
@@ -1041,6 +1220,7 @@ vaid||valid
 vaild||valid
 valide||valid
 variantions||variations
+varible||variable
 varient||variant
 vaule||value
 verbse||verbose
@@ -1053,7 +1233,9 @@ virtaul||virtual
 virtiual||virtual
 visiters||visitors
 vitual||virtual
+wakeus||wakeups
 wating||waiting
+wiat||wait
 wether||whether
 whataver||whatever
 whcih||which