diff options
Diffstat (limited to 'binary_search_tool')
34 files changed, 419 insertions, 299 deletions
diff --git a/binary_search_tool/MAINTENANCE b/binary_search_tool/MAINTENANCE index 8e5b3c61..8f96ff10 100644 --- a/binary_search_tool/MAINTENANCE +++ b/binary_search_tool/MAINTENANCE @@ -1,3 +1,7 @@ +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + This document is for future maintainers of the binary search/bisection tools. Authors: @@ -21,7 +25,9 @@ The following are good reference materials on how the tool works: TESTING: All unit tests live under the ./test directory. However, these tests -specifically test binary_search_state.py, binary_search_perforce.py, bisect.py. +specifically test binary_search_state.py, binary_search_perforce.py, +run_bisect.py. + These unit tests will not test the specific logic for ChromeOS/Android bisection. To test the ChromeOS/Android bisectors, use the common/hash_test.sh test. This is a simple test case that just checks the hashes of files on your @@ -62,10 +68,11 @@ Some of the design decisions are a bit difficult to understand from just reading the code unfortunately. I will attempt to clear up the major offenders of this: 1. common.py's argument dictionary: - binary_search_state.py and bisect.py both have to have near identical - arguments in order to support argument overriding in bisect.py. However - they do have to be slightly different. Mainly, bisect.py needs to have no - default values for arguments (so it can determine what's being overriden). + binary_search_state.py and run_bisect.py both have to have near identical + arguments in order to support argument overriding in run_bisect.py. However + they do have to be slightly different. Mainly, run_bisect.py needs to have + no default values for arguments (so it can determine what's being + overriden). In order to reduce huge amounts of code duplication for the argument building, we put argument building in common.py. That way both modules diff --git a/binary_search_tool/README.bisect.md b/binary_search_tool/README.bisect.md index 74715ca0..bd9e0f14 100644 --- a/binary_search_tool/README.bisect.md +++ b/binary_search_tool/README.bisect.md @@ -1,6 +1,6 @@ -# `bisect.py` +# `run_bisect.py` -`bisect.py` is a wrapper around the general purpose +`run_bisect.py` is a wrapper around the general purpose `binary_search_state.py`. It provides a user friendly interface for bisecting various compilation errors. The 2 currently provided methods of bisecting are ChromeOS package and object bisection. Each @@ -33,8 +33,8 @@ pings the machine and prompts the user if the machine is good. /build/${board}.work - A full copy of /build/${board}.bad ``` -1. Cleanup: bisect.py does most cleanup for you, the only thing required by the - user is to cleanup all built images and the three build trees made in +1. Cleanup: run_bisect.py does most cleanup for you, the only thing required by + the user is to cleanup all built images and the three build trees made in `/build/` 1. Default Arguments: @@ -58,13 +58,13 @@ pings the machine and prompts the user if the machine is good. 1. Basic interactive test package bisection, on daisy board: ``` - ./bisect.py package daisy 172.17.211.184 + ./run_bisect.py package daisy 172.17.211.184 ``` 2. Basic boot test package bisection, on daisy board: ``` - ./bisect.py package daisy 172.17.211.184 -t cros_pkg/boot_test.sh + ./run_bisect.py package daisy 172.17.211.184 -t cros_pkg/boot_test.sh ``` ### ChromeOS Object @@ -125,10 +125,10 @@ pings the machine and prompts the user if the machine is good. 1. Examples: 1. Basic interactive test object bisection, on daisy board for cryptohome - package: `./bisect.py object daisy 172.17.211.184 cryptohome` + package: `./run_bisect.py object daisy 172.17.211.184 cryptohome` 2. Basic boot test package bisection, on daisy board for cryptohome - package: `./bisect.py object daisy 172.17.211.184 cryptohome + package: `./run_bisect.py object daisy 172.17.211.184 cryptohome --test_script=sysroot_wrapper/boot_test.sh` ### Android object @@ -202,16 +202,17 @@ the following arguments: 1. Examples: 1. Basic interactive test android bisection, where the android source is at - ~/android_src: `./bisect.py android ~/android_src` + ~/android_src: `./run_bisect.py android ~/android_src` 2. Basic boot test android bisection, where the android source is at - `~/android_src`, and 10 jobs will be used to build android: `./bisect.py + `~/android_src`, and 10 jobs will be used to build android: + `./run_bisect.py android ~/android_src --num_jobs=10 --test_script=sysroot_wrapper/boot_test.sh` ### Resuming -`bisect.py` and `binary_search_state.py` offer the +`run_bisect.py` and `binary_search_state.py` offer the ability to resume a bisection in case it was interrupted by a SIGINT, power failure, etc. Every time the tool completes a bisection iteration its state is saved to disk (usually to the file @@ -221,14 +222,14 @@ completed iteration. ### Overriding -You can run `./bisect.py --help` or `./binary_search_state.py +You can run `./run_bisect.py --help` or `./binary_search_state.py --help` for a full list of arguments that can be overriden. Here are a couple of examples: Example 1 (do boot test instead of interactive test): ``` -./bisect.py package daisy 172.17.211.182 --test_script=cros_pkg/boot_test.sh +./run_bisect.py package daisy 172.17.211.182 --test_script=cros_pkg/boot_test.sh ``` Example 2 (do package bisector system test instead of interactive test, this @@ -236,13 +237,13 @@ Example 2 (do package bisector system test instead of interactive test, this hash_test.sh for more details): ``` -./bisect.py package daisy 172.17.211.182 \ +./run_bisect.py package daisy 172.17.211.182 \ --test_script=common/hash_test.sh --test_setup_script="" ``` Example 3 (enable verbose mode, disable pruning, and disable verification): ``` -./bisect.py package daisy 172.17.211.182 +./run_bisect.py package daisy 172.17.211.182 --verbose --prune=False --verify=False ``` diff --git a/binary_search_tool/README.pass_bisect.md b/binary_search_tool/README.pass_bisect.md index d03a563b..eb0f12a6 100644 --- a/binary_search_tool/README.pass_bisect.md +++ b/binary_search_tool/README.pass_bisect.md @@ -41,7 +41,7 @@ them. bisection: ``` - ./bisect.py android PATH_TO_ANDROID_HOME_DIR + ./run_bisect.py android PATH_TO_ANDROID_HOME_DIR --pass_bisect=’android/generate_cmd.sh’ --prune=False --ir_diff diff --git a/binary_search_tool/README.testing.md b/binary_search_tool/README.testing.md index 139dc087..0f06679b 100644 --- a/binary_search_tool/README.testing.md +++ b/binary_search_tool/README.testing.md @@ -28,7 +28,7 @@ To run the basic unit tests: ``` $ cd toolchain-utils/binary_search_tool/test -$ ./binary_search_tool_tester.py +$ ./binary_search_tool_test.py ``` # Running the bisection tests, testing the compiler wrapper. @@ -59,7 +59,7 @@ Move to the correct directory, then run the test script: ``` $ cd toolchain-utils/binary_search_tool -$ ./run_bisect_test.py +$ ./run_bisect_tests.py ``` If you are testing with the CHROMEOS COMPILER WRAPPER, you MUST run the @@ -82,7 +82,7 @@ $ export PYTHONPATH=`pwd`:${PYTHONPATH} Run the test script: ``` -$ ./run_bisect_test.py +$ ./run_bisect_tests.py ``` diff --git a/binary_search_tool/__init__.py b/binary_search_tool/__init__.py index 8b137891..76500def 100644 --- a/binary_search_tool/__init__.py +++ b/binary_search_tool/__init__.py @@ -1 +1,4 @@ - +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. diff --git a/binary_search_tool/binary_search_perforce.py b/binary_search_tool/binary_search_perforce.py index a4f8c1c6..5065f709 100755 --- a/binary_search_tool/binary_search_perforce.py +++ b/binary_search_tool/binary_search_perforce.py @@ -1,9 +1,11 @@ -#!/usr/bin/env python2 - -# Copyright 2018 The Chromium OS Authors. All rights reserved. +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. + """Module of binary serch for perforce.""" +from __future__ import division from __future__ import print_function import math @@ -25,7 +27,7 @@ def _GetP4ClientSpec(client_name, p4_paths): if ' ' not in p4_path: p4_string += ' -a %s' % p4_path else: - p4_string += " -a \"" + (' //' + client_name + '/').join(p4_path) + "\"" + p4_string += ' -a "' + (' //' + client_name + '/').join(p4_path) + '"' return p4_string @@ -50,8 +52,8 @@ def GetP4Command(client_name, p4_port, p4_paths, checkoutdir, p4_snapshot=''): command += ' && cd ' + checkoutdir command += ' && cp ${HOME}/.p4config .' command += ' && chmod u+w .p4config' - command += " && echo \"P4PORT=" + p4_port + "\" >> .p4config" - command += " && echo \"P4CLIENT=" + client_name + "\" >> .p4config" + command += ' && echo "P4PORT=' + p4_port + '" >> .p4config' + command += ' && echo "P4CLIENT=' + client_name + '" >> .p4config' command += (' && g4 client ' + _GetP4ClientSpec(client_name, p4_paths)) command += ' && g4 sync ' command += ' && cd -' @@ -84,7 +86,7 @@ class BinarySearcherForPass(object): # For the first run, update self.hi with total pass/transformation count if self.hi == 0: self.hi = self.total - self.current = (self.hi + self.lo) / 2 + self.current = (self.hi + self.lo) // 2 message = ('Bisecting between: (%d, %d)' % (self.lo, self.hi)) self.logger.LogOutput(message, print_to_console=verbose) message = ('Current limit number: %d' % self.current) @@ -100,11 +102,11 @@ class BinarySearcherForPass(object): If status == 1, it means that runtime error still happens with current pass/ transformation, so we need to decrease upper bound for binary search. - Return: + Returns: True if we find the bad pass/transformation, or cannot find bad one after decreasing to the first pass/transformation. Otherwise False. """ - assert status == 0 or status == 1 or status == 125 + assert status in (0, 1, 125), status if self.current == 0: message = ('Runtime error occurs before first pass/transformation. ' @@ -145,7 +147,7 @@ class BinarySearcher(object): self.logger = logger.GetLogger() def SetSortedList(self, sorted_list): - assert len(sorted_list) > 0 + assert sorted_list self.sorted_list = sorted_list self.index_log = [] self.hi = len(sorted_list) - 1 @@ -159,7 +161,7 @@ class BinarySearcher(object): message = ('Revision: %s index: %d returned: %d' % (self.sorted_list[self.current], self.current, status)) self.logger.LogOutput(message, print_to_console=verbose) - assert status == 0 or status == 1 or status == 125 + assert status in (0, 1, 125), status self.index_log.append(self.current) self.status_log.append(status) bsp = BinarySearchPoint(self.sorted_list[self.current], status, tag) @@ -168,13 +170,13 @@ class BinarySearcher(object): if status == 125: self.skipped_indices.append(self.current) - if status == 0 or status == 1: + if status in (0, 1): if status == 0: self.lo = self.current + 1 elif status == 1: self.hi = self.current self.logger.LogOutput('lo: %d hi: %d\n' % (self.lo, self.hi)) - self.current = (self.lo + self.hi) / 2 + self.current = (self.lo + self.hi) // 2 if self.lo == self.hi: message = ('Search complete. First bad version: %s' @@ -194,19 +196,19 @@ class BinarySearcher(object): def GetNextFlakyBinary(self): t = (self.lo, self.current, self.hi) q = [t] - while len(q): + while q: element = q.pop(0) if element[1] in self.skipped_indices: # Go top - to_add = (element[0], (element[0] + element[1]) / 2, element[1]) + to_add = (element[0], (element[0] + element[1]) // 2, element[1]) q.append(to_add) # Go bottom - to_add = (element[1], (element[1] + element[2]) / 2, element[2]) + to_add = (element[1], (element[1] + element[2]) // 2, element[2]) q.append(to_add) else: self.current = element[1] return - assert len(q), 'Queue should never be 0-size!' + assert q, 'Queue should never be 0-size!' def GetNextFlakyLinear(self): current_hi = self.current @@ -225,7 +227,7 @@ class BinarySearcher(object): current_lo -= 1 def GetNext(self): - self.current = (self.hi + self.lo) / 2 + self.current = (self.hi + self.lo) // 2 # Try going forward if current is skipped. if self.current in self.skipped_indices: self.GetNextFlakyBinary() @@ -295,7 +297,7 @@ class VCSBinarySearcher(object): def GetNextRevision(self): pass - def CheckoutRevision(self, revision): + def CheckoutRevision(self, current_revision): pass def SetStatus(self, status): @@ -362,14 +364,14 @@ class P4BinarySearcher(VCSBinarySearcher): if not os.path.isfile(self.checkout_dir + '/.p4config'): command = 'cd %s' % self.checkout_dir command += ' && cp ${HOME}/.p4config .' - command += " && echo \"P4PORT=" + self.p4_port + "\" >> .p4config" - command += " && echo \"P4CLIENT=" + self.client_name + "\" >> .p4config" + command += ' && echo "P4PORT=' + self.p4_port + '" >> .p4config' + command += ' && echo "P4CLIENT=' + self.client_name + '" >> .p4config' self.ce.RunCommand(command) command = 'cd %s' % self.checkout_dir command += '; g4 changes -c %s' % self.client_name _, out, _ = self.ce.RunCommandWOUTPUOT(command) changes = re.findall(r'Change (\d+)', out) - if len(changes) != 0: + if changes: command = 'cd %s' % self.checkout_dir for change in changes: command += '; g4 revert -c %s' % change @@ -413,9 +415,8 @@ class P4GCCBinarySearcher(P4BinarySearcher): for pr in problematic_ranges: if cr in range(pr[0], pr[1]): patch_file = '/home/asharif/triage_tool/%d-%d.patch' % (pr[0], pr[1]) - f = open(patch_file) - patch = f.read() - f.close() + with open(patch_file, encoding='utf-8') as f: + patch = f.read() files = re.findall('--- (//.*)', patch) command += '; cd %s' % self.checkout_dir for f in files: @@ -505,7 +506,7 @@ def Main(argv): logger.GetLogger().LogOutput('Cleaning up...') finally: logger.GetLogger().LogOutput(str(p4gccbs.bs), print_to_console=verbose) - status = p4gccbs.Cleanup() + p4gccbs.Cleanup() if __name__ == '__main__': diff --git a/binary_search_tool/binary_search_state.py b/binary_search_tool/binary_search_state.py index f6c8ac7c..1ddd65ce 100755 --- a/binary_search_tool/binary_search_state.py +++ b/binary_search_tool/binary_search_state.py @@ -1,10 +1,12 @@ -#!/usr/bin/env python2 - -# Copyright 2018 The Chromium OS Authors. All rights reserved. +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. + """The binary search wrapper.""" +from __future__ import division from __future__ import print_function import argparse @@ -20,15 +22,14 @@ import tempfile import time # Adds cros_utils to PYTHONPATH -import common +from binary_search_tool import binary_search_perforce +from binary_search_tool import common +from binary_search_tool import pass_mapping # Now we do import from cros_utils from cros_utils import command_executer from cros_utils import logger -import binary_search_perforce -import pass_mapping - GOOD_SET_VAR = 'BISECT_GOOD_SET' BAD_SET_VAR = 'BISECT_BAD_SET' @@ -37,11 +38,6 @@ HIDDEN_STATE_FILE = os.path.join( os.path.dirname(STATE_FILE), '.%s' % os.path.basename(STATE_FILE)) -class Error(Exception): - """The general binary search tool error class.""" - pass - - @contextlib.contextmanager def SetFile(env_var, items): """Generate set files that can be used by switch/test scripts. @@ -59,7 +55,7 @@ def SetFile(env_var, items): env_var: What environment variable to store the file name in. items: What items are in this set. """ - with tempfile.NamedTemporaryFile() as f: + with tempfile.NamedTemporaryFile('w', encoding='utf-8') as f: os.environ[env_var] = f.name f.write('\n'.join(items)) f.flush() @@ -100,10 +96,10 @@ class BinarySearchState(object): self.cmd_script = None self.mode = None self.PopulateItemsUsingCommand(self.get_initial_items) - self.currently_good_items = set([]) - self.currently_bad_items = set([]) - self.found_items = set([]) - self.known_good = set([]) + self.currently_good_items = set() + self.currently_bad_items = set() + self.found_items = set() + self.known_good = set() self.start_time = time.time() @@ -161,7 +157,7 @@ class BinarySearchState(object): item_list: list of all items to be switched """ if self.file_args: - with tempfile.NamedTemporaryFile() as f: + with tempfile.NamedTemporaryFile('w', encoding='utf-8') as f: f.write('\n'.join(item_list)) f.flush() command = '%s %s' % (switch_script, f.name) @@ -174,9 +170,8 @@ class BinarySearchState(object): command, print_to_console=self.verbose) except OSError as e: if e.errno == errno.E2BIG: - raise Error('Too many arguments for switch script! Use --file_args') - else: - raise + raise RuntimeError('Too many arguments for switch script! Use ' + '--file_args') assert ret == 0, 'Switch script %s returned %d' % (switch_script, ret) def TestScript(self): @@ -349,12 +344,12 @@ class BinarySearchState(object): return None def BuildWithPassLimit(self, limit, generate_ir=False): - """ Rebuild bad item with pass level bisect limit + """Rebuild bad item with pass level bisect limit Run command line script generated by GenerateBadCommandScript(), with pass level limit flags. - Return: + Returns: pass_num: current number of the pass, or total number of passes if limit set to -1. pass_name: The debugcounter name of current limit pass. @@ -389,7 +384,7 @@ class BinarySearchState(object): break pass_num += 1 last_pass = l - if limit != -1 and pass_num != limit: + if limit not in (-1, pass_num): raise ValueError('[Error] While building, limit number does not match.') return pass_num, self.CollectPassName(last_pass) @@ -398,16 +393,19 @@ class BinarySearchState(object): pass_name=None, pass_limit=-1, generate_ir=False): - """ Rebuild bad item with transformation level bisect limit + """Rebuild bad item with transformation level bisect limit Run command line script generated by GenerateBadCommandScript(), with pass level limit flags and transformation level limit flags. Args: - limit: transformation level limit for bad item - pass_name: name of bad pass debugcounter from pass level bisect result - pass_limit: pass level limit from pass level bisect result - Return: Total number of transformations if limit set to -1, else return 0. + limit: transformation level limit for bad item. + pass_name: name of bad pass debugcounter from pass level bisect result. + pass_limit: pass level limit from pass level bisect result. + generate_ir: Whether to generate IR comparison. + + Returns: + Total number of transformations if limit set to -1, else return 0. """ counter_name = pass_name @@ -513,7 +511,7 @@ class BinarySearchState(object): 'Total %s number: %d' % (self.mode, self.binary_search.total)) trans_index, _ = self.DoBinarySearchBadPass(pass_index, pass_name) - if (trans_index == 0): + if trans_index == 0: raise ValueError('Bisecting %s cannot reproduce good result.' % pass_name) if self.ir_diff: @@ -530,13 +528,15 @@ class BinarySearchState(object): Args: pass_index: Works for transformation level bisection, indicates the limit - number of pass from pass level bisecting result. + number of pass from pass level bisecting result. pass_name: Works for transformation level bisection, indicates - DebugCounter name of the bad pass from pass level bisecting result. - Return: + DebugCounter name of the bad pass from pass level bisecting + result. + + Returns: index: Index of problematic pass/transformation. pass_name: Works for pass level bisection, returns DebugCounter name for - bad pass. + bad pass. """ # If in resume mode don't reset search_cycles if not self.resumed: @@ -609,7 +609,7 @@ class BinarySearchState(object): new data. Raises: - Error if STATE_FILE already exists but is not a symlink. + OSError if STATE_FILE already exists but is not a symlink. """ ce, l = self.ce, self.l self.ce, self.l, self.binary_search.logger = None, None, None @@ -623,8 +623,8 @@ class BinarySearchState(object): if os.path.islink(STATE_FILE): old_state = os.readlink(STATE_FILE) else: - raise Error(('%s already exists and is not a symlink!\n' - 'State file saved to %s' % (STATE_FILE, path))) + raise OSError(('%s already exists and is not a symlink!\n' + 'State file saved to %s' % (STATE_FILE, path))) # Create new link and atomically overwrite old link temp_link = '%s.link' % HIDDEN_STATE_FILE @@ -642,26 +642,27 @@ class BinarySearchState(object): if not os.path.isfile(STATE_FILE): return None try: - bss = pickle.load(file(STATE_FILE)) - bss.l = logger.GetLogger() - bss.ce = command_executer.GetCommandExecuter() - bss.binary_search.logger = bss.l - bss.start_time = time.time() - - # Set resumed to be True so we can enter DoBinarySearch without the method - # resetting our current search_cycles to 0. - bss.resumed = True - - # Set currently_good_items and currently_bad_items to empty so that the - # first iteration after resuming will always be non-incremental. This is - # just in case the environment changes, the user makes manual changes, or - # a previous switch_script corrupted the environment. - bss.currently_good_items = set([]) - bss.currently_bad_items = set([]) - - binary_search_perforce.verbose = bss.verbose - return bss - except StandardError: + with open(STATE_FILE, 'rb') as f: + bss = pickle.load(f) + bss.l = logger.GetLogger() + bss.ce = command_executer.GetCommandExecuter() + bss.binary_search.logger = bss.l + bss.start_time = time.time() + + # Set resumed to be True so we can enter DoBinarySearch without the + # method resetting our current search_cycles to 0. + bss.resumed = True + + # Set currently_good_items and currently_bad_items to empty so that the + # first iteration after resuming will always be non-incremental. This + # is just in case the environment changes, the user makes manual + # changes, or a previous switch_script corrupted the environment. + bss.currently_good_items = set() + bss.currently_bad_items = set() + + binary_search_perforce.verbose = bss.verbose + return bss + except Exception: return None def RemoveState(self): @@ -686,8 +687,8 @@ class BinarySearchState(object): """Return h m s format of elapsed time since execution has started.""" diff = int(time.time() - self.start_time) seconds = diff % 60 - minutes = (diff / 60) % 60 - hours = diff / (60 * 60) + minutes = (diff // 60) % 60 + hours = diff // (60 * 60) seconds = str(seconds).rjust(2) minutes = str(minutes).rjust(2) @@ -871,16 +872,12 @@ def Run(get_initial_items, verify, file_args, verbose) bss.DoVerify() - try: - bss.DoSearchBadItems() - if pass_bisect: - bss.DoSearchBadPass() - bss.RemoveState() - logger.GetLogger().LogOutput( - 'Total execution time: %s' % bss.ElapsedTimeString()) - except Error as e: - logger.GetLogger().LogError(e) - return 1 + bss.DoSearchBadItems() + if pass_bisect: + bss.DoSearchBadPass() + bss.RemoveState() + logger.GetLogger().LogOutput( + 'Total execution time: %s' % bss.ElapsedTimeString()) return 0 diff --git a/binary_search_tool/bisect_driver.py b/binary_search_tool/bisect_driver.py index 82df15d6..ac37ad9f 100644 --- a/binary_search_tool/bisect_driver.py +++ b/binary_search_tool/bisect_driver.py @@ -1,10 +1,10 @@ -# Copyright 2016 Googie Inc. All rights Reserved. +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. # # This script is used to help the compiler wrapper in the ChromeOS and # Android build systems bisect for bad object files. -# -# pylint: disable=not-callable -# pylint: disable=indentation """Utilities for bisection of ChromeOS and Android object files. @@ -267,10 +267,10 @@ def cache_file(execargs, bisect_dir, cache, abs_file_path): if CONTINUE_ON_REDUNDANCY: return True raise Exception( - 'Trying to cache file %s multiple times. To avoid the error, set \ - CONTINUE_ON_REDUNDANCY to 1. For reference, the list of such files \ - will be written to %s' % (abs_file_path, - os.path.join(population_dir, '_DUPS'))) + 'Trying to cache file %s multiple times. To avoid the error, set ' \ + 'BISECT_CONTINUE_ON_REDUNDANCY to 1. For reference, the list of ' \ + 'such files will be written to %s' % (abs_file_path, os.path.join( + population_dir, '_DUPS'))) shutil.copy2(abs_file_path, bisect_path) # Set cache object to be read-only so later compilations can't @@ -278,7 +278,8 @@ def cache_file(execargs, bisect_dir, cache, abs_file_path): os.chmod(bisect_path, 0o444) return True else: - # File not found (happens when compilation fails but error code is still 0) + # File not found (happens when compilation fails but error code is still + # 0) return False except Exception: print('Could not cache file %s' % abs_file_path, file=sys.stderr) diff --git a/binary_search_tool/common.py b/binary_search_tool/common.py index 40660b52..85cd478b 100644 --- a/binary_search_tool/common.py +++ b/binary_search_tool/common.py @@ -1,4 +1,5 @@ -# Copyright 2018 The Chromium OS Authors. All rights reserved. +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. @@ -7,14 +8,14 @@ This module serves two main purposes: 1. Programatically include the utils module in PYTHONPATH 2. Create the argument parsing shared between binary_search_state.py and - bisect.py + run_bisect.py The argument parsing is handled by populating _ArgsDict with all arguments. -_ArgsDict is required so that binary_search_state.py and bisect.py can share -the argument parsing, but treat them slightly differently. For example, -bisect.py requires that all argument defaults are suppressed so that overriding -can occur properly (i.e. only options that are explicitly entered by the user -end up in the resultant options dictionary). +_ArgsDict is required so that binary_search_state.py and run_bisect.py can +share the argument parsing, but treat them slightly differently. For example, +run_bisect.py requires that all argument defaults are suppressed so that +overriding can occur properly (i.e. only options that are explicitly entered +by the user end up in the resultant options dictionary). ArgumentDict inherits OrderedDict in order to preserve the order the args are created so the help text is made properly. @@ -32,8 +33,8 @@ if os.path.isabs(sys.argv[0]): os.path.dirname(sys.argv[0]))) else: wdir = os.getcwd() - utils_pythonpath = os.path.abspath('{0}/{1}/..'.format(wdir, os.path.dirname( - sys.argv[0]))) + utils_pythonpath = os.path.abspath('{0}/{1}/..'.format( + wdir, os.path.dirname(sys.argv[0]))) sys.path.append(utils_pythonpath) @@ -55,8 +56,10 @@ class ArgumentDict(collections.OrderedDict): ... } """ - _POSSIBLE_OPTIONS = ['action', 'nargs', 'const', 'default', 'type', 'choices', - 'required', 'help', 'metavar', 'dest'] + _POSSIBLE_OPTIONS = [ + 'action', 'nargs', 'const', 'default', 'type', 'choices', 'required', + 'help', 'metavar', 'dest' + ] def AddArgument(self, *args, **kwargs): """Add argument to ArgsDict, has same signature as argparse.add_argument @@ -77,7 +80,7 @@ class ArgumentDict(collections.OrderedDict): TypeError: if args is empty or if option in kwargs is not a valid option for argparse.add_argument. """ - if len(args) == 0: + if not args: raise TypeError('Argument needs at least one name') for key in kwargs: @@ -106,8 +109,8 @@ def BuildArgParser(parser, override=False): Args: parser: type argparse.ArgumentParser, will call add_argument for every item in _ArgsDict - override: True if being called from bisect.py. Used to say that default and - required options are to be ignored + override: True if being called from run_bisect.py. Used to say that default + and required options are to be ignored Returns: None @@ -115,7 +118,7 @@ def BuildArgParser(parser, override=False): ArgsDict = GetArgsDict() # Have no defaults when overriding - for arg_names, arg_options in ArgsDict.iteritems(): + for arg_names, arg_options in ArgsDict.items(): if override: arg_options = arg_options.copy() arg_options.pop('default', None) @@ -147,34 +150,34 @@ def _BuildArgsDict(args): '--get_initial_items', dest='get_initial_items', help='Script to run to get the initial objects. ' - 'If your script requires user input ' - 'the --verbose option must be used') + 'If your script requires user input ' + 'the --verbose option must be used') args.AddArgument( '-g', '--switch_to_good', dest='switch_to_good', help='Script to run to switch to good. ' - 'If your switch script requires user input ' - 'the --verbose option must be used') + 'If your switch script requires user input ' + 'the --verbose option must be used') args.AddArgument( '-b', '--switch_to_bad', dest='switch_to_bad', help='Script to run to switch to bad. ' - 'If your switch script requires user input ' - 'the --verbose option must be used') + 'If your switch script requires user input ' + 'the --verbose option must be used') args.AddArgument( '-I', '--test_setup_script', dest='test_setup_script', help='Optional script to perform building, flashing, ' - 'and other setup before the test script runs.') + 'and other setup before the test script runs.') args.AddArgument( '-t', '--test_script', dest='test_script', help='Script to run to test the ' - 'output after packages are built.') + 'output after packages are built.') # No input (evals to False), # --prune (evals to True), # --prune=False, @@ -189,19 +192,19 @@ def _BuildArgsDict(args): type=StrToBool, metavar='bool', help='If True, continue until all bad items are found. ' - 'Defaults to False.') + 'Defaults to False.') args.AddArgument( '-P', '--pass_bisect', dest='pass_bisect', default=None, help='Script to generate another script for pass level bisect, ' - 'which contains command line options to build bad item. ' - 'This will also turn on pass/transformation level bisection. ' - 'Needs support of `-opt-bisect-limit`(pass) and ' - '`-print-debug-counter`(transformation) from LLVM. ' - 'For now it only supports one single bad item, so to use it, ' - 'prune must be set to False.') + 'which contains command line options to build bad item. ' + 'This will also turn on pass/transformation level bisection. ' + 'Needs support of `-opt-bisect-limit`(pass) and ' + '`-print-debug-counter`(transformation) from LLVM. ' + 'For now it only supports one single bad item, so to use it, ' + 'prune must be set to False.') # No input (evals to False), # --ir_diff (evals to True), # --ir_diff=False, @@ -216,8 +219,8 @@ def _BuildArgsDict(args): type=StrToBool, metavar='bool', help='Whether to print IR differences before and after bad ' - 'pass/transformation to verbose output. Defaults to False, ' - 'only works when pass_bisect is enabled.') + 'pass/transformation to verbose output. Defaults to False, ' + 'only works when pass_bisect is enabled.') # No input (evals to False), # --noincremental (evals to True), # --noincremental=False, @@ -231,8 +234,8 @@ def _BuildArgsDict(args): default=False, type=StrToBool, metavar='bool', - help='If True, don\'t propagate good/bad changes ' - 'incrementally. Defaults to False.') + help="If True, don't propagate good/bad changes " + 'incrementally. Defaults to False.') # No input (evals to False), # --file_args (evals to True), # --file_args=False, @@ -247,7 +250,7 @@ def _BuildArgsDict(args): type=StrToBool, metavar='bool', help='Whether to use a file to pass arguments to scripts. ' - 'Defaults to False.') + 'Defaults to False.') # No input (evals to True), # --verify (evals to True), # --verify=False, @@ -261,7 +264,7 @@ def _BuildArgsDict(args): type=StrToBool, metavar='bool', help='Whether to run verify iterations before searching. ' - 'Defaults to True.') + 'Defaults to True.') args.AddArgument( '-N', '--prune_iterations', @@ -289,5 +292,5 @@ def _BuildArgsDict(args): dest='resume', action='store_true', help='Resume bisection tool execution from state file.' - 'Useful if the last bisection was terminated ' - 'before it could properly finish.') + 'Useful if the last bisection was terminated ' + 'before it could properly finish.') diff --git a/binary_search_tool/compiler_wrapper.py b/binary_search_tool/compiler_wrapper.py index a6d189b5..8755ed42 100755 --- a/binary_search_tool/compiler_wrapper.py +++ b/binary_search_tool/compiler_wrapper.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Prototype compiler wrapper. Only tested with: gcc, g++, clang, clang++ @@ -21,7 +26,7 @@ import os import shlex import sys -import bisect_driver +from binary_search_tool import bisect_driver WRAPPED = '%s.real' % sys.argv[0] BISECT_STAGE = os.environ.get('BISECT_STAGE') diff --git a/binary_search_tool/cros_pkg/create_cleanup_script.py b/binary_search_tool/cros_pkg/create_cleanup_script.py index ed4eab61..62ee38f1 100755 --- a/binary_search_tool/cros_pkg/create_cleanup_script.py +++ b/binary_search_tool/cros_pkg/create_cleanup_script.py @@ -1,6 +1,9 @@ -#!/usr/bin/env python2 -# -# Copyright 2015 Google Inc. All Rights Reserved +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """The script to generate a cleanup script after setup.sh. This script takes a set of flags, telling it what setup.sh changed @@ -69,20 +72,22 @@ def Main(argv): if options.old_link or options.renamed_tree: if not options.tree_existed: - Usage(parser, 'If --tree_existed is False, cannot have ' - '--renamed_tree or --old_link') + Usage( + parser, 'If --tree_existed is False, cannot have ' + '--renamed_tree or --old_link') if options.old_link and options.renamed_tree: Usage(parser, '--old_link and --renamed_tree are incompatible options.') if options.tree_existed: if not options.old_link and not options.renamed_tree: - Usage(parser, 'If --tree_existed is True, then must have either ' - '--old_link or --renamed_tree') + Usage( + parser, 'If --tree_existed is True, then must have either ' + '--old_link or --renamed_tree') out_filename = 'cros_pkg/' + options.board + '_cleanup.sh' - with open(out_filename, 'w') as out_file: + with open(out_filename, 'w', encoding='utf-8') as out_file: out_file.write('#!/bin/bash\n\n') # First, remove the 'new' soft link. out_file.write('sudo rm /build/%s\n' % options.board) @@ -100,8 +105,8 @@ def Main(argv): original_link = original_link[1:] if original_link[-1] == "'": original_link = original_link[:-1] - out_file.write('sudo ln -s %s /build/%s\n' % (original_link, - options.board)) + out_file.write( + 'sudo ln -s %s /build/%s\n' % (original_link, options.board)) out_file.write('\n') # Remove common.sh file out_file.write('rm common/common.sh\n') diff --git a/binary_search_tool/pass_mapping.py b/binary_search_tool/pass_mapping.py index cb80910c..2678fd6d 100644 --- a/binary_search_tool/pass_mapping.py +++ b/binary_search_tool/pass_mapping.py @@ -1,6 +1,8 @@ -# Copyright 2018 The Chromium OS Authors. All rights reserved. +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. + """Config file for pass level bisection Provides a mapping from pass info from -opt-bisect result to DebugCounter name. diff --git a/binary_search_tool/bisect.py b/binary_search_tool/run_bisect.py index c7dd5238..ef1048bb 100755 --- a/binary_search_tool/bisect.py +++ b/binary_search_tool/run_bisect.py @@ -1,28 +1,29 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """The unified package/object bisecting tool.""" from __future__ import print_function import abc import argparse +from argparse import RawTextHelpFormatter import os import sys -from argparse import RawTextHelpFormatter -import common +from binary_search_tool import binary_search_state +from binary_search_tool import common from cros_utils import command_executer from cros_utils import logger -import binary_search_state - -class Bisector(object): +class Bisector(object, metaclass=abc.ABCMeta): """The abstract base class for Bisectors.""" - # Make Bisector an abstract class - __metaclass__ = abc.ABCMeta - def __init__(self, options, overrides=None): """Constructor for Bisector abstract base class @@ -44,7 +45,7 @@ class Bisector(object): which arguments have been overridden. Example output: - ./bisect.py package daisy 172.17.211.184 -I "" -t cros_pkg/my_test.sh + ./run_bisect.py package daisy 172.17.211.184 -I "" -t cros_pkg/my_test.sh Performing ChromeOS Package bisection Method Config: board : daisy @@ -83,10 +84,8 @@ class Bisector(object): out += '\nBisection Config: (* = overridden)\n' max_key_len = max([len(str(x)) for x in args.keys()]) # Print args in common._ArgsDict order - args_order = [x['dest'] for x in common.GetArgsDict().itervalues()] - compare = lambda x, y: cmp(args_order.index(x), args_order.index(y)) - - for key in sorted(args, cmp=compare): + args_order = [x['dest'] for x in common.GetArgsDict().values()] + for key in sorted(args, key=args_order.index): val = args[key] key_str = str(key).rjust(max_key_len) val_str = str(val) @@ -193,9 +192,9 @@ class BisectObject(Bisector): if options.dir: os.environ['BISECT_DIR'] = options.dir self.options.dir = os.environ.get('BISECT_DIR', '/tmp/sysroot_bisect') - self.setup_cmd = ('%s %s %s %s' % - (self.sysroot_wrapper_setup, self.options.board, - self.options.remote, self.options.package)) + self.setup_cmd = ( + '%s %s %s %s' % (self.sysroot_wrapper_setup, self.options.board, + self.options.remote, self.options.package)) self.ArgOverride(self.default_kwargs, overrides) @@ -306,7 +305,7 @@ def Run(bisector): _HELP_EPILOG = """ -Run ./bisect.py {method} --help for individual method help/args +Run ./run_bisect.py {method} --help for individual method help/args ------------------ @@ -320,7 +319,7 @@ def Main(argv): override_parser = argparse.ArgumentParser( add_help=False, argument_default=argparse.SUPPRESS, - usage='bisect.py {mode} [options]') + usage='run_bisect.py {mode} [options]') common.BuildArgParser(override_parser, override=True) epilog = _HELP_EPILOG + override_parser.format_help() diff --git a/binary_search_tool/run_bisect_test.py b/binary_search_tool/run_bisect_tests.py index d4ff4f73..9172d678 100755 --- a/binary_search_tool/run_bisect_test.py +++ b/binary_search_tool/run_bisect_tests.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Run full bisection test.""" from __future__ import print_function @@ -81,7 +86,7 @@ def verify_compiler_and_wrapper(): # We don't need to do any special setup if running inside a ChromeOS # chroot. if os.path.exists('/usr/bin/x86_64-cros-linux-gnu-gcc'): - return + return True message = """ *** IMPORTANT --- READ THIS CAREFULLY!! *** diff --git a/binary_search_tool/sysroot_wrapper/testing_test.py b/binary_search_tool/sysroot_wrapper/testing_test.py index a0d6ca10..b5ceec1f 100755 --- a/binary_search_tool/sysroot_wrapper/testing_test.py +++ b/binary_search_tool/sysroot_wrapper/testing_test.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Test for sysroot_wrapper bisector. All files in bad_files will be determined to be bad. This test was made for @@ -15,9 +20,9 @@ import os base_path = ('/var/cache/chromeos-chrome/chrome-src-internal/src/out_daisy/' 'Release/obj/') bad_files = [ - os.path.join(base_path, 'base/base.cpu.o'), os.path.join( - base_path, 'base/base.version.o'), os.path.join(base_path, - 'apps/apps.launcher.o') + os.path.join(base_path, 'base/base.cpu.o'), + os.path.join(base_path, 'base/base.version.o'), + os.path.join(base_path, 'apps/apps.launcher.o') ] bisect_dir = os.environ.get('BISECT_DIR', '/tmp/sysroot_bisect') diff --git a/binary_search_tool/test/__init__.py b/binary_search_tool/test/__init__.py index 8b137891..76500def 100644 --- a/binary_search_tool/test/__init__.py +++ b/binary_search_tool/test/__init__.py @@ -1 +1,4 @@ - +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. diff --git a/binary_search_tool/test/binary_search_tool_tester.py b/binary_search_tool/test/binary_search_tool_test.py index aff45a86..ca9313b5 100755 --- a/binary_search_tool/test/binary_search_tool_tester.py +++ b/binary_search_tool/test/binary_search_tool_test.py @@ -1,10 +1,12 @@ -#!/usr/bin/env python2 - -# Copyright 2018 The Chromium OS Authors. All rights reserved. +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. + """Tests for bisecting tool.""" +from __future__ import division from __future__ import print_function __author__ = 'shenhan@google.com (Han Shen)' @@ -16,15 +18,15 @@ import unittest from cros_utils import command_executer from binary_search_tool import binary_search_state -from binary_search_tool import bisect +from binary_search_tool import run_bisect -import common -import gen_obj +from binary_search_tool.test import common +from binary_search_tool.test import gen_obj def GenObj(): obj_num = random.randint(100, 1000) - bad_obj_num = random.randint(obj_num / 100, obj_num / 20) + bad_obj_num = random.randint(obj_num // 100, obj_num // 20) if bad_obj_num == 0: bad_obj_num = 1 gen_obj.Main(['--obj_num', str(obj_num), '--bad_obj_num', str(bad_obj_num)]) @@ -38,10 +40,10 @@ def CleanObj(): class BisectTest(unittest.TestCase): - """Tests for bisect.py""" + """Tests for run_bisect.py""" def setUp(self): - with open('./is_setup', 'w'): + with open('./is_setup', 'w', encoding='utf-8'): pass try: @@ -57,8 +59,8 @@ class BisectTest(unittest.TestCase): except OSError: pass - class FullBisector(bisect.Bisector): - """Test bisector to test bisect.py with""" + class FullBisector(run_bisect.Bisector): + """Test bisector to test run_bisect.py with""" def __init__(self, options, overrides): super(BisectTest.FullBisector, self).__init__(options, overrides) @@ -81,14 +83,14 @@ class BisectTest(unittest.TestCase): return 0 def test_full_bisector(self): - ret = bisect.Run(self.FullBisector({}, {})) - self.assertEquals(ret, 0) + ret = run_bisect.Run(self.FullBisector({}, {})) + self.assertEqual(ret, 0) self.assertFalse(os.path.exists(common.OBJECTS_FILE)) self.assertFalse(os.path.exists(common.WORKING_SET_FILE)) def check_output(self): _, out, _ = command_executer.GetCommandExecuter().RunCommandWOutput( - ('grep "Bad items are: " logs/binary_search_tool_tester.py.out | ' + ('grep "Bad items are: " logs/binary_search_tool_test.py.out | ' 'tail -n1')) ls = out.splitlines() self.assertEqual(len(ls), 1) @@ -113,7 +115,7 @@ class BisectingUtilsTest(unittest.TestCase): """Generate [100-1000] object files, and 1-5% of which are bad ones.""" GenObj() - with open('./is_setup', 'w'): + with open('./is_setup', 'w', encoding='utf-8'): pass try: @@ -146,7 +148,7 @@ class BisectingUtilsTest(unittest.TestCase): test_script='./is_good.py', prune=True, file_args=True) - self.assertEquals(ret, 0) + self.assertEqual(ret, 0) self.check_output() def test_arg_parse(self): @@ -156,7 +158,7 @@ class BisectingUtilsTest(unittest.TestCase): '--test_script', './is_good.py', '--prune', '--file_args' ] ret = binary_search_state.Main(args) - self.assertEquals(ret, 0) + self.assertEqual(ret, 0) self.check_output() def test_test_setup_script(self): @@ -178,7 +180,7 @@ class BisectingUtilsTest(unittest.TestCase): test_setup_script='./test_setup.py', prune=True, file_args=True) - self.assertEquals(ret, 0) + self.assertEqual(ret, 0) self.check_output() def test_bad_test_setup_script(self): @@ -196,15 +198,15 @@ class BisectingUtilsTest(unittest.TestCase): state_file = binary_search_state.STATE_FILE hidden_state_file = os.path.basename(binary_search_state.HIDDEN_STATE_FILE) - with open(state_file, 'w') as f: + with open(state_file, 'w', encoding='utf-8') as f: f.write('test123') bss = binary_search_state.MockBinarySearchState() with self.assertRaises(binary_search_state.Error): bss.SaveState() - with open(state_file, 'r') as f: - self.assertEquals(f.read(), 'test123') + with open(state_file, 'r', encoding='utf-8') as f: + self.assertEqual(f.read(), 'test123') os.remove(state_file) @@ -243,9 +245,9 @@ class BisectingUtilsTest(unittest.TestCase): bss = None bss2 = binary_search_state.MockBinarySearchState.LoadState() - self.assertEquals(bss2.all_items, test_items) - self.assertEquals(bss2.currently_good_items, set([])) - self.assertEquals(bss2.currently_bad_items, set([])) + self.assertEqual(bss2.all_items, test_items) + self.assertEqual(bss2.currently_good_items, set([])) + self.assertEqual(bss2.currently_bad_items, set([])) def test_tmp_cleanup(self): bss = binary_search_state.MockBinarySearchState( @@ -255,14 +257,14 @@ class BisectingUtilsTest(unittest.TestCase): bss.SwitchToGood(['0', '1', '2', '3']) tmp_file = None - with open('tmp_file', 'r') as f: + with open('tmp_file', 'r', encoding='utf-8') as f: tmp_file = f.read() os.remove('tmp_file') self.assertFalse(os.path.exists(tmp_file)) ws = common.ReadWorkingSet() for i in range(3): - self.assertEquals(ws[i], 42) + self.assertEqual(ws[i], 42) def test_verify_fail(self): bss = binary_search_state.MockBinarySearchState( @@ -298,11 +300,11 @@ class BisectingUtilsTest(unittest.TestCase): prune=False, file_args=True) bss.DoSearchBadItems() - self.assertEquals(len(bss.found_items), 1) + self.assertEqual(len(bss.found_items), 1) bad_objs = common.ReadObjectsFile() found_obj = int(bss.found_items.pop()) - self.assertEquals(bad_objs[found_obj], 1) + self.assertEqual(bad_objs[found_obj], 1) def test_set_file(self): binary_search_state.Run( @@ -326,12 +328,12 @@ class BisectingUtilsTest(unittest.TestCase): noincremental=True, file_args=True, verify=False) - self.assertEquals(ret, 0) + self.assertEqual(ret, 0) self.check_output() def check_output(self): _, out, _ = command_executer.GetCommandExecuter().RunCommandWOutput( - ('grep "Bad items are: " logs/binary_search_tool_tester.py.out | ' + ('grep "Bad items are: " logs/binary_search_tool_test.py.out | ' 'tail -n1')) ls = out.splitlines() self.assertEqual(len(ls), 1) @@ -354,7 +356,7 @@ class BisectingUtilsPassTest(BisectingUtilsTest): def check_pass_output(self, pass_name, pass_num, trans_num): _, out, _ = command_executer.GetCommandExecuter().RunCommandWOutput( - ('grep "Bad pass: " logs/binary_search_tool_tester.py.out | ' + ('grep "Bad pass: " logs/binary_search_tool_test.py.out | ' 'tail -n1')) ls = out.splitlines() self.assertEqual(len(ls), 1) @@ -365,7 +367,7 @@ class BisectingUtilsPassTest(BisectingUtilsTest): _, out, _ = command_executer.GetCommandExecuter().RunCommandWOutput( ('grep "Bad transformation number: ' - '" logs/binary_search_tool_tester.py.out | ' + '" logs/binary_search_tool_test.py.out | ' 'tail -n1')) ls = out.splitlines() self.assertEqual(len(ls), 1) @@ -383,7 +385,7 @@ class BisectingUtilsPassTest(BisectingUtilsTest): pass_bisect='./generate_cmd.py', prune=True, file_args=True) - self.assertEquals(ret, 1) + self.assertEqual(ret, 1) def test_gen_cmd_script(self): bss = binary_search_state.MockBinarySearchState( @@ -488,7 +490,7 @@ class BisectStressTest(unittest.TestCase): prune=True, file_args=True, verify=False) - self.assertEquals(ret, 0) + self.assertEqual(ret, 0) self.check_output() def test_every_index_is_bad(self): @@ -506,12 +508,12 @@ class BisectStressTest(unittest.TestCase): test_script='./is_good.py', prune=True, file_args=True) - self.assertEquals(ret, 0) + self.assertEqual(ret, 0) self.check_output() def check_output(self): _, out, _ = command_executer.GetCommandExecuter().RunCommandWOutput( - ('grep "Bad items are: " logs/binary_search_tool_tester.py.out | ' + ('grep "Bad items are: " logs/binary_search_tool_test.py.out | ' 'tail -n1')) ls = out.splitlines() self.assertEqual(len(ls), 1) diff --git a/binary_search_tool/test/cmd_script.py b/binary_search_tool/test/cmd_script.py index eb91fe9b..bfd56052 100755 --- a/binary_search_tool/test/cmd_script.py +++ b/binary_search_tool/test/cmd_script.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Command script without compiler support for pass level bisection. This script generates a pseudo log which a workable compiler should print out. @@ -11,7 +16,7 @@ from __future__ import print_function import os import sys -import common +from binary_search_tool.test import common def Main(argv): diff --git a/binary_search_tool/test/cmd_script_no_support.py b/binary_search_tool/test/cmd_script_no_support.py index a817f300..badbedc8 100644 --- a/binary_search_tool/test/cmd_script_no_support.py +++ b/binary_search_tool/test/cmd_script_no_support.py @@ -1,3 +1,8 @@ +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Command script without compiler support for pass level bisection. This script generates a pseudo log when certain bisecting flags are not @@ -13,8 +18,9 @@ import sys def Main(): if not os.path.exists('./is_setup'): return 1 - print('No support for -opt-bisect-limit or -print-debug-counter.', - file=sys.stderr) + print( + 'No support for -opt-bisect-limit or -print-debug-counter.', + file=sys.stderr) return 0 diff --git a/binary_search_tool/test/common.py b/binary_search_tool/test/common.py index 5c3ff538..cf5300f5 100755 --- a/binary_search_tool/test/common.py +++ b/binary_search_tool/test/common.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Common utility functions.""" DEFAULT_OBJECT_NUMBER = 1238 @@ -9,33 +14,29 @@ WORKING_SET_FILE = 'working_set.txt' def ReadWorkingSet(): working_set = [] - f = open(WORKING_SET_FILE, 'r') - for l in f: - working_set.append(int(l)) - f.close() + with open(WORKING_SET_FILE, 'r', encoding='utf-8') as f: + for l in f: + working_set.append(int(l)) return working_set def WriteWorkingSet(working_set): - f = open(WORKING_SET_FILE, 'w') - for o in working_set: - f.write('{0}\n'.format(o)) - f.close() + with open(WORKING_SET_FILE, 'w', encoding='utf-8') as f: + for o in working_set: + f.write('{0}\n'.format(o)) def ReadObjectsFile(): objects_file = [] - f = open(OBJECTS_FILE, 'r') - for l in f: - objects_file.append(int(l)) - f.close() + with open(OBJECTS_FILE, 'r', encoding='utf-8') as f: + for l in f: + objects_file.append(int(l)) return objects_file def ReadObjectIndex(filename): object_index = [] - f = open(filename, 'r') - for o in f: - object_index.append(int(o)) - f.close() + with open(filename, 'r', encoding='utf-8') as f: + for o in f: + object_index.append(int(o)) return object_index diff --git a/binary_search_tool/test/gen_init_list.py b/binary_search_tool/test/gen_init_list.py index 002fc352..bc5dd8fe 100755 --- a/binary_search_tool/test/gen_init_list.py +++ b/binary_search_tool/test/gen_init_list.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Prints out index for every object file, starting from 0.""" from __future__ import print_function @@ -6,7 +11,7 @@ from __future__ import print_function import sys from cros_utils import command_executer -import common +from binary_search_tool.test import common def Main(): diff --git a/binary_search_tool/test/gen_obj.py b/binary_search_tool/test/gen_obj.py index a2bc7d93..4f65c71b 100755 --- a/binary_search_tool/test/gen_obj.py +++ b/binary_search_tool/test/gen_obj.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Script to generate a list of object files. 0 represents a good object file. @@ -12,7 +17,7 @@ import os import random import sys -import common +from binary_search_tool.test import common def Main(argv): @@ -77,12 +82,11 @@ def Main(argv): if os.path.isfile(common.WORKING_SET_FILE): os.remove(common.WORKING_SET_FILE) - f = open(common.OBJECTS_FILE, 'w') - w = open(common.WORKING_SET_FILE, 'w') - for i in obj_list: - f.write('{0}\n'.format(i)) - w.write('{0}\n'.format(i)) - f.close() + with open(common.OBJECTS_FILE, 'w', encoding='utf-8') as f: + with open(common.WORKING_SET_FILE, 'w', encoding='utf-8') as w: + for i in obj_list: + f.write('{0}\n'.format(i)) + w.write('{0}\n'.format(i)) obj_num = len(obj_list) bad_obj_num = obj_list.count(1) diff --git a/binary_search_tool/test/generate_cmd.py b/binary_search_tool/test/generate_cmd.py index f6876eda..51b36b0a 100755 --- a/binary_search_tool/test/generate_cmd.py +++ b/binary_search_tool/test/generate_cmd.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Generate a virtual cmd script for pass level bisection. This is a required argument for pass level bisecting. For unit test, we use @@ -15,7 +20,7 @@ def Main(): if not os.path.exists('./is_setup'): return 1 file_name = 'cmd_script.sh' - with open(file_name, 'w') as f: + with open(file_name, 'w', encoding='utf-8') as f: f.write('Generated by generate_cmd.py') return 0 diff --git a/binary_search_tool/test/is_good.py b/binary_search_tool/test/is_good.py index a0be4a08..662921e8 100755 --- a/binary_search_tool/test/is_good.py +++ b/binary_search_tool/test/is_good.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Check to see if the working set produces a good executable.""" from __future__ import print_function @@ -6,7 +11,7 @@ from __future__ import print_function import os import sys -import common +from binary_search_tool.test import common def Main(): diff --git a/binary_search_tool/test/is_good_noinc_prune.py b/binary_search_tool/test/is_good_noinc_prune.py index a900bd32..c0e42bb1 100755 --- a/binary_search_tool/test/is_good_noinc_prune.py +++ b/binary_search_tool/test/is_good_noinc_prune.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Check to see if the working set produces a good executable. This test script is made for the noincremental-prune test. This makes sure @@ -12,16 +17,16 @@ from __future__ import print_function import os import sys -import common +from binary_search_tool.test import common def Main(): working_set = common.ReadWorkingSet() - with open('noinc_prune_good', 'r') as good_args: + with open('noinc_prune_good', 'r', encoding='utf-8') as good_args: num_good_args = len(good_args.readlines()) - with open('noinc_prune_bad', 'r') as bad_args: + with open('noinc_prune_bad', 'r', encoding='utf-8') as bad_args: num_bad_args = len(bad_args.readlines()) num_args = num_good_args + num_bad_args diff --git a/binary_search_tool/test/switch_tmp.py b/binary_search_tool/test/switch_tmp.py index 51b7110e..0f3c4234 100755 --- a/binary_search_tool/test/switch_tmp.py +++ b/binary_search_tool/test/switch_tmp.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Change portions of the object files to good. This file is a test switch script. Used only for the test test_tmp_cleanup. @@ -11,7 +16,7 @@ from __future__ import print_function import sys -import common +from binary_search_tool.test import common def Main(argv): @@ -23,7 +28,7 @@ def Main(argv): working_set[int(oi)] = 42 common.WriteWorkingSet(working_set) - with open('tmp_file', 'w') as f: + with open('tmp_file', 'w', encoding='utf-8') as f: f.write(argv[1]) return 0 diff --git a/binary_search_tool/test/switch_to_bad.py b/binary_search_tool/test/switch_to_bad.py index a1b6bd59..e3553eb6 100755 --- a/binary_search_tool/test/switch_to_bad.py +++ b/binary_search_tool/test/switch_to_bad.py @@ -1,11 +1,16 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Switch part of the objects file in working set to (possible) bad ones.""" from __future__ import print_function import sys -import common +from binary_search_tool.test import common def Main(argv): diff --git a/binary_search_tool/test/switch_to_bad_noinc_prune.py b/binary_search_tool/test/switch_to_bad_noinc_prune.py index db76acad..81b558e1 100755 --- a/binary_search_tool/test/switch_to_bad_noinc_prune.py +++ b/binary_search_tool/test/switch_to_bad_noinc_prune.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Switch part of the objects file in working set to (possible) bad ones. The "portion" is defined by the file (which is passed as the only argument to @@ -18,7 +23,7 @@ from __future__ import print_function import shutil import sys -import common +from binary_search_tool.test import common def Main(argv): diff --git a/binary_search_tool/test/switch_to_bad_set_file.py b/binary_search_tool/test/switch_to_bad_set_file.py index edf226d3..5b941c62 100755 --- a/binary_search_tool/test/switch_to_bad_set_file.py +++ b/binary_search_tool/test/switch_to_bad_set_file.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Switch part of the objects file in working set to (possible) bad ones. This script is meant to be specifically used with the set_file test. This uses @@ -10,7 +15,7 @@ from __future__ import print_function import os import sys -import common +from binary_search_tool.test import common def Main(_): diff --git a/binary_search_tool/test/switch_to_good.py b/binary_search_tool/test/switch_to_good.py index 59a118c1..97479329 100755 --- a/binary_search_tool/test/switch_to_good.py +++ b/binary_search_tool/test/switch_to_good.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Change portions of the object files to good. The "portion" is defined by the file (which is passed as the only argument to @@ -10,7 +15,7 @@ from __future__ import print_function import sys -import common +from binary_search_tool.test import common def Main(argv): diff --git a/binary_search_tool/test/switch_to_good_noinc_prune.py b/binary_search_tool/test/switch_to_good_noinc_prune.py index 00488a74..0b91a0d8 100755 --- a/binary_search_tool/test/switch_to_good_noinc_prune.py +++ b/binary_search_tool/test/switch_to_good_noinc_prune.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Change portions of the object files to good. The "portion" is defined by the file (which is passed as the only argument to @@ -18,7 +23,7 @@ from __future__ import print_function import shutil import sys -import common +from binary_search_tool.test import common def Main(argv): diff --git a/binary_search_tool/test/switch_to_good_set_file.py b/binary_search_tool/test/switch_to_good_set_file.py index b5e521f9..1cb05e0c 100755 --- a/binary_search_tool/test/switch_to_good_set_file.py +++ b/binary_search_tool/test/switch_to_good_set_file.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Change portions of the object files to good. The "portion" is defined by the file (which is passed as the only argument to @@ -14,7 +19,7 @@ from __future__ import print_function import os import sys -import common +from binary_search_tool.test import common def Main(_): diff --git a/binary_search_tool/test/test_setup.py b/binary_search_tool/test/test_setup.py index 0d6a410e..ecc8eb97 100755 --- a/binary_search_tool/test/test_setup.py +++ b/binary_search_tool/test/test_setup.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Emulate running of test setup script, is_good.py should fail without this.""" from __future__ import print_function @@ -8,7 +13,7 @@ import sys def Main(): # create ./is_setup - with open('./is_setup', 'w'): + with open('./is_setup', 'w', encoding='utf-8'): pass return 0 diff --git a/binary_search_tool/test/test_setup_bad.py b/binary_search_tool/test/test_setup_bad.py index d715f57a..cbca3c21 100755 --- a/binary_search_tool/test/test_setup_bad.py +++ b/binary_search_tool/test/test_setup_bad.py @@ -1,4 +1,9 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# Copyright 2020 The Chromium OS Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + """Emulate test setup that fails (i.e. failed flash to device)""" from __future__ import print_function |