[bisection tool] Fix bisect_driver.py and udpate for host bisection.

This CL udpates bisect_driver.py to consolidate all the various versions
we have into one "source of truth" bisect_driver.py.  It also updates
bisect_driver.py to ignore .o files generated during the 'configure' stage
of Makefile/Autoconf packages (such as GCC & LLVM).  And it adds two
scripts to help with host package bisection.

BUG=chromium:697995
TEST=Successfully ran unit tests; successfully bisected LLVM inside ChromeOS.

Change-Id: I2e766299dadf9bf3c7de6496f507c863d446d00d
Reviewed-on: https://chromium-review.googlesource.com/696031
Commit-Ready: Caroline Tice <cmtice@chromium.org>
Tested-by: Caroline Tice <cmtice@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

4 files changed, 173 insertions, 41 deletions

diff --git a/binary_search_tool/README.bisect b/binary_search_tool/README.bisect
index e6185e8a..49e0c085 100644
--- a/binary_search_tool/README.bisect
+++ b/binary_search_tool/README.bisect
@@ -1,10 +1,11 @@
 
-bisect.py is a wrapper around the general purpose binary_search_state.py. It
-provides a user friendly interface for bisecting various compilation errors.
-The 2 currently provided methods of bisecting are ChromeOS package and object
-bisection. Each method defines a default set of options to pass to
-binary_search_state.py and allow the user to override these defaults (see
-the "Overriding" section).
+bisect.py is a wrapper around the general purpose
+binary_search_state.py. It provides a user friendly interface for
+bisecting various compilation errors.  The 2 currently provided
+methods of bisecting are ChromeOS package and object bisection. Each
+method defines a default set of options to pass to
+binary_search_state.py and allow the user to override these defaults
+(see the "Overriding" section).
 
 ** NOTE **
 All commands, examples, scripts, etc. are to be run from your chroot unless
@@ -31,8 +32,9 @@ Bisection Methods:
       /build/${board}.work - A full copy of /build/${board}.bad
 
   b) Cleanup:
-    bisect.py does most cleanup for you, the only thing required by the user is
-    to cleanup all built images and the three build trees made in /build/
+    bisect.py does most cleanup for you, the only
+    thing required by the user is to cleanup all built images and the
+    three build trees made in /build/
 
   c) Default Arguments:
     --get_initial_items='cros_pkg/get_initial_items.sh'
@@ -187,16 +189,18 @@ Bisection Methods:
         --test_script=sysroot_wrapper/boot_test.sh
 
 Resuming:
-  bisect.py and binary_search_state.py offer the ability to resume a bisection
-  in case it was interrupted by a SIGINT, power failure, etc. Every time the
-  tool completes a bisection iteration its state is saved to disk (usually to
-  the file "./bisect.py.state"). If passed the --resume option, the tool
+  bisect.py and binary_search_state.py offer the
+  ability to resume a bisection in case it was interrupted by a
+  SIGINT, power failure, etc. Every time the tool completes a
+  bisection iteration its state is saved to disk (usually to the file
+  "./bisect_driver.py.state"). If passed the --resume option, the tool
   it will automatically detect the state file and resume from the last
   completed iteration.
 
 Overriding:
-  You can run ./bisect.py --help or ./binary_search_state.py --help for a full
-  list of arguments that can be overriden. Here are a couple of examples:
+  You can run ./bisect.py --help or ./binary_search_state.py
+  --help for a full list of arguments that can be overriden. Here are
+  a couple of examples:
 
   Example 1 (do boot test instead of interactive test):
   ./bisect.py package daisy 172.17.211.182 --test_script=cros_pkg/boot_test.sh
diff --git a/binary_search_tool/bisect_driver.py b/binary_search_tool/bisect_driver.py
index 0b3fb1d4..21dd11fa 100644
--- a/binary_search_tool/bisect_driver.py
+++ b/binary_search_tool/bisect_driver.py
@@ -1,8 +1,8 @@
-# Copyright 2016 Google Inc. All Rights Reserved.
+# Copyright 2016 Googie Inc.  All rights Reserved.
 #
-# This script is used to help the compiler wrapper in the Android build system
-# bisect for bad object files.
-"""Utilities for bisection of Android object files.
+# This script is used to help the compiler wrapper in the ChromeOS and
+# Android build systems bisect for bad object files.
+"""Utilities for bisection of ChromeOS and Android object files.
 
 This module contains a set of utilities to allow bisection between
 two sets (good and bad) of object files. Mostly used to find compiler
@@ -24,7 +24,7 @@ import shutil
 import subprocess
 import sys
 
-VALID_MODES = ['POPULATE_GOOD', 'POPULATE_BAD', 'TRIAGE']
+VALID_MODES = ('POPULATE_GOOD', 'POPULATE_BAD', 'TRIAGE')
 GOOD_CACHE = 'good'
 BAD_CACHE = 'bad'
 LIST_FILE = os.path.join(GOOD_CACHE, '_LIST')
@@ -35,7 +35,6 @@ WRAPPER_SAFE_MODE = os.environ.get('BISECT_WRAPPER_SAFE_MODE', None) == '1'
 
 class Error(Exception):
   """The general compiler wrapper error class."""
-  pass
 
 
 @contextlib.contextmanager
@@ -58,7 +57,13 @@ def lock_file(path, mode):
     mode: mode to open file with ('w', 'r', etc.)
   """
   with open(path, mode) as f:
-    # Share the lock if just reading, make lock exclusive if writing
+    # Apply FD_CLOEXEC argument to fd. This ensures that the file descriptor
+    # won't be leaked to any child processes.
+    current_args = fcntl.fcntl(f.fileno(), fcntl.F_GETFD)
+    fcntl.fcntl(f.fileno(), fcntl.F_SETFD, current_args | fcntl.FD_CLOEXEC)
+
+    # Reads can share the lock as no race conditions exist. If write is needed,
+    # give writing process exclusive access to the file.
     if f.mode == 'r' or f.mode == 'rb':
       lock_type = fcntl.LOCK_SH
     else:
@@ -68,8 +73,6 @@ def lock_file(path, mode):
       fcntl.lockf(f, lock_type)
       yield f
       f.flush()
-    except:
-      raise
     finally:
       fcntl.lockf(f, fcntl.LOCK_UN)
 
@@ -103,8 +106,7 @@ def which_cache(obj_file):
   determine where an object file should be linked from (good or bad).
   """
   bad_set_file = os.environ.get('BISECT_BAD_SET')
-  ret = subprocess.call(['grep', '-x', '-q', obj_file, bad_set_file])
-  if ret == 0:
+  if in_object_list(obj_file, bad_set_file):
     return BAD_CACHE
   else:
     return GOOD_CACHE
@@ -124,18 +126,29 @@ def get_obj_path(execargs):
 
   Returns:
     Absolute object path from execution args (-o argument). If no object being
-    outputted or output doesn't end in ".o" then return empty string.
+    outputted, then return empty string. -o argument is checked only if -c is
+    also present.
   """
   try:
     i = execargs.index('-o')
+    _ = execargs.index('-c')
   except ValueError:
     return ''
 
   obj_path = execargs[i + 1]
-  if not obj_path.endswith(('.o',)):
-    # TODO: what suffixes do we need to contemplate
-    # TODO: add this as a warning
-    # TODO: need to handle -r compilations
+  # Ignore args that do not create a file.
+  if obj_path in (
+      '-',
+      '/dev/null',):
+    return ''
+  # Ignore files ending in .tmp.
+  if obj_path.endswith(('.tmp',)):
+    return ''
+  # Ignore configuration files generated by Automake/Autoconf/CMake etc.
+  if (obj_path.endswith('conftest.o') or
+      obj_path.endswith('CMakeFiles/test.o') or
+      obj_path.find('CMakeTmp') != -1 or
+      os.path.abspath(obj_path).find('CMakeTmp') != -1):
     return ''
 
   return os.path.abspath(obj_path)
@@ -151,7 +164,7 @@ def get_dep_path(execargs):
   if '-MD' not in execargs and '-MMD' not in execargs:
     return ''
 
-  # If -MF given this is the path of the dependency file. Otherwise the
+  # If -MF is given this is the path of the dependency file. Otherwise the
   # dependency file is the value of -o but with a .d extension
   if '-MF' in execargs:
     i = execargs.index('-MF')
@@ -217,7 +230,16 @@ def get_side_effects(execargs):
 
 
 def cache_file(execargs, bisect_dir, cache, abs_file_path):
-  """Cache compiler output file (.o/.d/.dwo)."""
+  """Cache compiler output file (.o/.d/.dwo).
+
+  Args:
+    execargs: compiler execution arguments.
+    bisect_dir: The directory where bisection caches live.
+    cache: Which cache the file will be cached to (GOOD/BAD).
+    abs_file_path: Absolute path to file being cached.
+  Returns:
+    True if caching was successful, False otherwise.
+  """
   # os.path.join fails with absolute paths, use + instead
   bisect_path = os.path.join(bisect_dir, cache) + abs_file_path
   bisect_path_dir = os.path.dirname(bisect_path)
@@ -227,14 +249,36 @@ def cache_file(execargs, bisect_dir, cache, abs_file_path):
 
   try:
     if os.path.exists(abs_file_path):
+      if os.path.exists(bisect_path):
+        # File exists
+        population_dir = os.path.join(bisect_dir, cache)
+        with lock_file(os.path.join(population_dir, '_DUPS'),
+                       'a') as dup_object_list:
+          dup_object_list.write('%s\n' % abs_file_path)
+        raise Exception(
+            'Trying to cache file %s multiple times.' % abs_file_path)
+
       shutil.copy2(abs_file_path, bisect_path)
+      # Set cache object to be read-only so later compilations can't
+      # accidentally overwrite it.
+      os.chmod(bisect_path, 0o444)
+      return True
+    else:
+      # File not found (happens when compilation fails but error code is still 0)
+      return False
   except Exception:
     print('Could not cache file %s' % abs_file_path, file=sys.stderr)
     raise
 
 
 def restore_file(bisect_dir, cache, abs_file_path):
-  """Restore file from cache (.o/.d/.dwo)."""
+  """Restore file from cache (.o/.d/.dwo).
+
+  Args:
+    bisect_dir: The directory where bisection caches live.
+    cache: Which cache the file will be restored from (GOOD/BAD).
+    abs_file_path: Absolute path to file being restored.
+  """
   # os.path.join fails with absolute paths, use + instead
   cached_path = os.path.join(bisect_dir, cache) + abs_file_path
   if os.path.exists(cached_path):
@@ -264,21 +308,41 @@ def bisect_populate(execargs, bisect_dir, population_name):
     return retval
 
   full_obj_path = get_obj_path(execargs)
-  # If not a normal compiler call then just exit
+  # This is not a normal compiler call because it doesn't have a -o argument,
+  # or the -o argument has an unusable output file.
+  # It's likely that this compiler call was actually made to invoke the linker,
+  # or as part of a configuratoin test. In this case we want to simply call the
+  # compiler and return.
   if not full_obj_path:
-    return
+    return retval
 
-  cache_file(execargs, bisect_dir, population_name, full_obj_path)
+  # Return if not able to cache the object file
+  if not cache_file(execargs, bisect_dir, population_name, full_obj_path):
+    return retval
 
   population_dir = os.path.join(bisect_dir, population_name)
   with lock_file(os.path.join(population_dir, '_LIST'), 'a') as object_list:
     object_list.write('%s\n' % full_obj_path)
 
   for side_effect in get_side_effects(execargs):
-    cache_file(execargs, bisect_dir, population_name, side_effect)
+    _ = cache_file(execargs, bisect_dir, population_name, side_effect)
+
+  return retval
 
 
 def bisect_triage(execargs, bisect_dir):
+  """Use object object file from appropriate cache (good/bad).
+
+  Given a populated bisection directory, use the object file saved
+  into one of the caches (good/bad) according to what is specified
+  in the good/bad sets. The good/bad sets are generated by the
+  high level binary search tool. Additionally restore any possible
+  side effects of compiler.
+
+  Args:
+    execargs: compiler execution arguments.
+    bisect_dir: populated bisection directory.
+  """
   full_obj_path = get_obj_path(execargs)
   obj_list = os.path.join(bisect_dir, LIST_FILE)
 
@@ -309,7 +373,7 @@ def bisect_triage(execargs, bisect_dir):
       return retval
     os.remove(full_obj_path)
     restore_file(bisect_dir, cache, full_obj_path)
-    return
+    return retval
 
   # Generate compiler side effects. Trick Make into thinking compiler was
   # actually executed.
@@ -321,14 +385,16 @@ def bisect_triage(execargs, bisect_dir):
   if not os.path.exists(full_obj_path):
     restore_file(bisect_dir, cache, full_obj_path)
 
+  return 0
+
 
 def bisect_driver(bisect_stage, bisect_dir, execargs):
   """Call appropriate bisection stage according to value in bisect_stage."""
   if bisect_stage == 'POPULATE_GOOD':
-    bisect_populate(execargs, bisect_dir, GOOD_CACHE)
+    return bisect_populate(execargs, bisect_dir, GOOD_CACHE)
   elif bisect_stage == 'POPULATE_BAD':
-    bisect_populate(execargs, bisect_dir, BAD_CACHE)
+    return bisect_populate(execargs, bisect_dir, BAD_CACHE)
   elif bisect_stage == 'TRIAGE':
-    bisect_triage(execargs, bisect_dir)
+    return bisect_triage(execargs, bisect_dir)
   else:
     raise ValueError('wrong value for BISECT_STAGE: %s' % bisect_stage)
diff --git a/binary_search_tool/sysroot_wrapper/interactive_test_host.sh b/binary_search_tool/sysroot_wrapper/interactive_test_host.sh
new file mode 100755
index 00000000..58adffc0
--- /dev/null
+++ b/binary_search_tool/sysroot_wrapper/interactive_test_host.sh
@@ -0,0 +1,25 @@
+#!/bin/bash -u
+#
+# Copyright 2017 Google Inc. All Rights Reserved.
+#
+# This script is intended to be used by binary_search_state.py, as
+# part of the binary search triage on ChromeOS package and object files for a
+# host package. It waits for the test setup script to build the image, then asks
+# the user if the image is good or not. (Since this is a host package, there is
+# no 'install' phase needed.)  This script should return '0' if the test succeeds
+# (the image is 'good'); '1' if the test fails (the image is 'bad'); and '125'
+# if it could not determine (does not apply in this case).
+#
+
+source common/common.sh
+
+while true; do
+    read -p "Is this a good ChromeOS image?" yn
+    case $yn in
+        [Yy]* ) exit 0;;
+        [Nn]* ) exit 1;;
+        * ) echo "Please answer yes or no.";;
+    esac
+done
+
+exit 125
diff --git a/binary_search_tool/sysroot_wrapper/test_setup_host.sh b/binary_search_tool/sysroot_wrapper/test_setup_host.sh
new file mode 100755
index 00000000..b5169eee
--- /dev/null
+++ b/binary_search_tool/sysroot_wrapper/test_setup_host.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+#
+# Copyright 2017 Google Inc. All Rights Reserved.
+#
+# This is a generic ChromeOS package/image test setup script. It is meant to
+# be used for either the object file or package bisection tools. This script
+# is intended to be used with host object bisection, to bisect the object
+# files in a host package.  Since it deals with a host package, there is no
+# building an image or flashing a device -- just building the host package
+# itself.
+#
+# This script is intended to be used by binary_search_state.py, as
+# part of the binary search triage on ChromeOS objects and packages. It should
+# return '0' if the setup succeeds; and '1' if the setup fails (the image
+# could not build or be flashed).
+#
+
+export PYTHONUNBUFFERED=1
+
+source common/common.sh
+
+
+if [[ "${BISECT_MODE}" == "OBJECT_MODE" ]]; then
+  echo "EMERGING ${BISECT_PACKAGE}"
+  sudo -E emerge ${BISECT_PACKAGE}
+  emerge_status=$?
+
+  if [[ ${emerge_status} -ne 0 ]] ; then
+    echo "emerging ${BISECT_PACKAGE} returned a non-zero status: $emerge_status"
+    exit 1
+  fi
+
+  exit 0
+fi
+
+
+exit 0