diff options
| author | Wei Wang <wvw@google.com> | 2020-08-05 13:26:00 -0700 |
|---|---|---|
| committer | Wei Wang <wvw@google.com> | 2020-08-05 13:34:28 -0700 |
| commit | 8be73e4796dddc52e141edcee353a7e7f1d07021 (patch) | |
| tree | 8a0d6fd78eb610653f12ded770595c80f1e89a61 /catapult/devil | |
| parent | 706249e9ec9e10c6c48508b7f84315104a3ff990 (diff) | |
| download | chromium-trace-8be73e4796dddc52e141edcee353a7e7f1d07021.tar.gz | |
Update catapult to latest version (6dacd170b)
git log --oneline --no-merges cad35e22d..6dacd170b tracing systrace
6dacd170b trace-viewer: Add polyfill to systrace viewer
8af044eac [v8][wasm] Introduce metric for compilation tierup
ade0fed37 metrics: Add diagnostics to console error metric
59d6160bf [v8][wasm] Centralize collection of events for wasm metrics
9a983c5e0 [ftrace importer] Add thermal parser
49d9f039e [v8][wasm] Introduce metric for baseline streaming compilation
4dd7f16b5 catapult: Rename black_list/white_list in presubmit
a207ab60c trace_viewer: Make links open in new tab
72669fe03 telemetry: Allow benchmarks to add an info blurb
0c87b3242 heap: Add metric for time that cannot be moved to concurrent thread
7bb44abcc [trace-viewer] Fix custom filename when saving traces
2e401eea3 [rendering] Report only the average for thread-times metrics.
32ec2ce70 [v8] Add first wasm metric for instantiation
de622ae39 [rendering] Remove breakdowns for cpu-time metrics.
c4dd7047a atrace_agent: Support /sys/kernel/tracing
e6867ba7c Peak Gpu Memory Ablation Telemetry
fbbd9ca81 rendering: Fix alerts for dropped frames.
9d178021d [tracing] Add Ampere-hour unit to histogram proto format
2755b192a Add more helpful error message when unit is not found
69e9be866 [tracing] Add Ampere-hour unit
f773e05e5 Remove the pipeline metrics and visualization charts
e5247fccd tracing: Fix suggested save filename for protobuf traces
4e00dc571 tracing: Add rudimentary protobuf trace support
03e7ff403 Fix events clashing
cd6894f0a Fix slice interleaving bug
1e4bfa556 gc_metric: Add scavenger stack survival metrics
06f14d9d4 Reland "Reland "Refractor out ts_proxy_server and webpagereplay_go_server to py_utils.""
eda74a743 [Tracing] Support None value in _variance in RunningStatistics
c5ffb2104 Revert "Reland "Refractor out ts_proxy_server and webpagereplay_go_server to py_utils.""
cd56abf06 Reland "Refractor out ts_proxy_server and webpagereplay_go_server to py_utils."
f00e7afd3 Added new tracing category presets
d06fd11a1 Revert "Refractor out ts_proxy_server and webpagereplay_go_server to py_utils."
7e5acd6c9 Refractor out ts_proxy_server and webpagereplay_go_server to py_utils.
953cd08e7 [metrics] Implement Total Blocking Time metric
ee5bb39ff [tracing] Change TTI metric to use FCP as start point
297fffa49 Add origin trial token for tests
e9a5bd129 rendering: Add a metric for total cpu/wall time ratio.
27d7ac2c1 rendering: Add a missing param.
3999e2615 rendering benchmark: Some no-op cleanups.
c72a211d0 Update the default systrace tags to the recommended Android defaults.
910f6c87a [catapult] Update message loop start time event name.
504782cdb [trace-viewer] Speed up rendering
1be7dc685 [trace-viewer] Speed up rendering (part 2)
a0519c673 [systrace] Change the count for new HTML format
438ea30dc [systrace] Add android_cgroup_agent
a8bbccaab Remove histogram proto JSON converter code.
8351d2182 [Catapult] Remove myself from OWNERS.
4a1f482cf [tracing] Ignore renderers without mainThread when computing metrics
bdb8301ab Remove some prints I accidentally left in.
6ff57586c Implement proto writing in the dashboard.
9ca9ff382 Support disabled-by-default-toplevel.flow to toplevel.flow rename
42d02fb84 [trace viewer] Speed up selection
a58084c62 Make all values in the proto doubles.
786ed18d9 Move the histogram proto stub to proto/.
7004f998c Fix parsing of summary options.
329b47817 Remove the proto test driver.
27182f007 Add a small proto->JSON tool.
7ce0bb097 trace_data: Raise exceptions only after traces are added to artifacts
21c99c447 Use doubles for the histogram C++ API.
e12edec0d Add method for setting summary options.
50d86304a [trace viewer] Add way to hide the polyfill warning
cc6ba530a [trace viewer] Update embedder doc and example
0503911de heap: Restructure blink gc metrics
657c0f420 [trace viewer] Fix test runner under polyfill
0d19767dc [results.html] Add webcomponents v0 polyfill to results.html
3b3069e9b [trace viewer] Fix a number of broken features under polyfill
cb90cdd77 [trace viewer] Fix keyboard navigation under polyfill
cc6825061 [trace viewer] Enable Webcomponents V0 Polyfill
601fc962a Add origin trial token for 127.0.0.1:8003
ea794b87a Teaching histograms to deserialize from protos.
3403d9601 Remove UE bucketing in runtimeStatsTotalMetric
0b016536a trace_data: fix typo in error message
9769628aa Use auto instead of google::protobuf in the histogram C++ API.
b9cc8d75a Fix gn gen --check breakage on the unit test target.
c8ebd366b Move to protobuf lite.
5546abd6b Fix bugs where I assume some optional fields were required.
d4405431b Add support for diagnostics and unit conversion.
fada3b97d Add reserved_infos.h/cc.
dd7cdd9b5 Use python 2 for trace2html
6a138fbe7 Add JSON converter and proto test driver.
13d8a54a7 heap: Adding metrics for Oilpan foreground/background marking time
32c9791b8 trace event import: Don't try to clock-translate duration values.
ae4bbcda1 Avoid infinite loop in updateMajorMarkData
85c4a438f Add support for sending histogram.proto JSON to /add_histograms.
debe1986c Add makefile for histogram.proto.
9506caf2e Fixed speed index computation in case of zero visible area.
69f98e5ab Implement Histogram C++ API for samples, and running statistics.
b9acce749 Add rect based speed index back to loading metric.
47e7bed70 Add proto definition for the new Histograms C++ API.
903bda890 Correct LCP metric use of Map.
69cc04af2 Add navigationStart histogram to loading_metric.
a61151f11 [tracing] Add khokhlov to relevant OWNERS files
731aba0ef Add a new category for shared memory in v8
8412598cc Add python inputs for tracing:generate_about_tracing
536c641b7 Update V8 runtime_stats to use different pattern for optimize background
9413ab06f Add origin trial token for local dev server
f34a9d2df Add --results-label command line option to run_metric
25db8d2fb Fix breakpad symbolization of heap profiles when so is mapped from apk.
6e71a5063 [tracing] Rename 'tir_label' to 'grouping_label'.
204286b69 Make HistogramBinBoundaries creation thread-safe
28ef213de Allow filtering flow events
d039ea0c1 [tracing/ui] Truncate long thread names instead of showing ellipsis
4b1db19bd trace-viewer: Avoid generating tracks when a process is hidden.
a38631cd7 Log JS errors on the server
bf69ed0d3 uma-metric: Specify the bins for peak gpu memory usage metric.
376ccc06c trace-viewer: Make some style changes for the toolbar.
c4191602b [tracing] Add HistogramSet.Merge method
e62dfe906 Add support for WebRTC internal commit ranges.
69077a26c Add ms to legacy unit infos
dd2bd9ef0 [metrics] Add metrics reported by page
a8bbe3b25 rendering: Remove some dead code.
54ac3634e Handle traces with process labels stripped from the metadata
eb8f5aa22 Remove main_thread_scroll_latency metric
c979465c5 Symbolize object type with frame name if enable "--frame-as-object-type"
77ad3c478 Add more owners for symbolizer script.
f2c3c6a6a [metrics] Move filtering of GC metrics to the metrics code
9b593a340 [Tracing] Add Main-Thread histograms to runtimeStatsTotalMetric.
8e42ececc [Tracing] Remove runtimeStatsMetric
3f0606454 [tracing] Replace crouleau@ with perezju@ for owning TBMv2 and tracing tooling.
5d664a903 Fix order of timestamps
dfd000a38 Add dproy as owner of tracing
900b41d09 [trace-viewer] Add support for viewing images
d7d4ef32e Remove visually complete by some time metric
ab8067161 Update viewport used in metric calculation
b38fb99e3 Update to match new traced values naming
35d417023 Remove speed index from loading metric
3b2499a6a Adds dproy as tracing/metrics owner
e46910861 [Telemetry] Add file name extensions to trace files
8bd58eb92 Update which screenshots are considered in calculating the metric
Bug: 162963924
Bug: 160818586
Test: external/chromium-trace/catapult/systrace/bin/systrace -o ~/trace.html sched freq gfx hal idle power workq irq view camera thermal -b 32768
Change-Id: Ie9f01eed98f8ab80e041777202f135dae4f8646e
Diffstat (limited to 'catapult/devil')
129 files changed, 8065 insertions, 5248 deletions
diff --git a/catapult/devil/.style.yapf b/catapult/devil/.style.yapf new file mode 100644 index 00000000..95091f08 --- /dev/null +++ b/catapult/devil/.style.yapf @@ -0,0 +1,7 @@ +[style] +based_on_style = pep8 + +allow_multiline_lambdas = True +column_limit = 80 +indent_dictionary_value = True +indent_width = 2 diff --git a/catapult/devil/PRESUBMIT.py b/catapult/devil/PRESUBMIT.py index 289a5c65..0b49eb80 100644 --- a/catapult/devil/PRESUBMIT.py +++ b/catapult/devil/PRESUBMIT.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Presubmit script for devil. See http://dev.chromium.org/developers/how-tos/depottools/presubmit-scripts for @@ -10,39 +9,42 @@ details on the presubmit API built into depot_tools. def _RunPylint(input_api, output_api): - return input_api.RunTests(input_api.canned_checks.RunPylint( - input_api, output_api, pylintrc='pylintrc')) + return input_api.RunTests( + input_api.canned_checks.RunPylint( + input_api, output_api, pylintrc='pylintrc')) def _RunUnitTests(input_api, output_api): def J(*dirs): """Returns a path relative to presubmit directory.""" - return input_api.os_path.join( - input_api.PresubmitLocalPath(), 'devil', *dirs) + return input_api.os_path.join(input_api.PresubmitLocalPath(), 'devil', + *dirs) test_env = dict(input_api.environ) test_env.update({ - 'PYTHONDONTWRITEBYTECODE': '1', - 'PYTHONPATH': ':'.join([J(), J('..')]), + 'PYTHONDONTWRITEBYTECODE': '1', + 'PYTHONPATH': ':'.join([J(), J('..')]), }) - message_type = (output_api.PresubmitError if input_api.is_committing - else output_api.PresubmitPromptWarning) + message_type = (output_api.PresubmitError if input_api.is_committing else + output_api.PresubmitPromptWarning) return input_api.RunTests([ input_api.Command( name='devil/bin/run_py_tests', cmd=[ - input_api.os_path.join( - input_api.PresubmitLocalPath(), 'bin', 'run_py_tests')], + input_api.os_path.join(input_api.PresubmitLocalPath(), 'bin', + 'run_py_tests') + ], kwargs={'env': test_env}, - message=message_type)]) + message=message_type) + ]) def _EnsureNoPylibUse(input_api, output_api): def other_python_files(f): - this_presubmit_file = input_api.os_path.join( - input_api.PresubmitLocalPath(), 'PRESUBMIT.py') + this_presubmit_file = input_api.os_path.join(input_api.PresubmitLocalPath(), + 'PRESUBMIT.py') return (f.LocalPath().endswith('.py') and not f.AbsoluteLocalPath() == this_presubmit_file) @@ -52,15 +54,16 @@ def _EnsureNoPylibUse(input_api, output_api): errors = [] for f in changed_files: - errors.extend( - '%s:%d' % (f.LocalPath(), line_number) - for line_number, line_text in f.ChangedContents() - if import_error_re.search(line_text)) + errors.extend('%s:%d' % (f.LocalPath(), line_number) + for line_number, line_text in f.ChangedContents() + if import_error_re.search(line_text)) if errors: - return [output_api.PresubmitError( - 'pylib modules should not be imported from devil modules.', - items=errors)] + return [ + output_api.PresubmitError( + 'pylib modules should not be imported from devil modules.', + items=errors) + ] return [] @@ -78,4 +81,3 @@ def CheckChangeOnUpload(input_api, output_api): def CheckChangeOnCommit(input_api, output_api): return CommonChecks(input_api, output_api) - diff --git a/catapult/devil/README.md b/catapult/devil/README.md index 9953e6ae..83f4ba09 100644 --- a/catapult/devil/README.md +++ b/catapult/devil/README.md @@ -27,11 +27,9 @@ devil also provides command-line utilities: ## Constraints and Caveats -devil is used with python 2.7. Its compatibility with python 3 has not been -tested, and neither achieving nor maintaining said compatibility is currently -a priority. +devil supports python 2.7. Python 3 compatibility is currently a work in +progress (see https://crbug.com/1007101). ## Contributing -Please see the [contributor's guide](https://github.com/catapult-project/catapult/blob/master/CONTRIBUTING.md). - +Please see [our contributor's guide](/CONTRIBUTING.md) diff --git a/catapult/devil/bin/generate_md_docs b/catapult/devil/bin/generate_md_docs index 634e14a5..1cca44c1 100755 --- a/catapult/devil/bin/generate_md_docs +++ b/catapult/devil/bin/generate_md_docs @@ -3,25 +3,25 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +import logging import os import sys -_DEVIL_PATH = os.path.abspath(os.path.join( - os.path.dirname(__file__), '..')) -_DEVIL_URL = ( - 'https://github.com/catapult-project/catapult/blob/master/devil/') +_DEVIL_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) +_DEVIL_URL = ('https://github.com/catapult-project/catapult/blob/master/devil/') sys.path.append(_DEVIL_PATH) from devil.utils import cmd_helper _FILES_TO_DOC = { - 'devil/android/sdk/adb_wrapper.py': 'docs/adb_wrapper.md', - 'devil/android/device_utils.py': 'docs/device_utils.md', - 'devil/utils/markdown.py': 'docs/markdown.md', + 'devil/android/sdk/adb_wrapper.py': 'docs/adb_wrapper.md', + 'devil/android/device_utils.py': 'docs/device_utils.md', + 'devil/utils/markdown.py': 'docs/markdown.md', } _MARKDOWN_SCRIPT = os.path.join(_DEVIL_PATH, 'devil', 'utils', 'markdown.py') + def main(): failed = False for k, v in _FILES_TO_DOC.iteritems(): @@ -29,9 +29,10 @@ def main(): module_link = _DEVIL_URL + k doc_path = os.path.join(_DEVIL_PATH, v) - status, stdout = cmd_helper.GetCmdStatusAndOutput( - [sys.executable, _MARKDOWN_SCRIPT, module_path, - '--module-link', module_link]) + status, stdout = cmd_helper.GetCmdStatusAndOutput([ + sys.executable, _MARKDOWN_SCRIPT, module_path, '--module-link', + module_link + ]) if status: logging.error('Failed to update doc for %s' % module_path) failed = True @@ -41,5 +42,6 @@ def main(): return 1 if failed else 0 + if __name__ == '__main__': sys.exit(main()) diff --git a/catapult/devil/bin/run_py_devicetests b/catapult/devil/bin/run_py_devicetests index 656bedf2..6a6da188 100755 --- a/catapult/devil/bin/run_py_devicetests +++ b/catapult/devil/bin/run_py_devicetests @@ -3,13 +3,13 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +import logging import os import sys -_CATAPULT_PATH = os.path.abspath(os.path.join( - os.path.dirname(__file__), '..', '..')) -_DEVIL_PATH = os.path.abspath(os.path.join( - os.path.dirname(__file__), '..')) +_CATAPULT_PATH = os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..')) +_DEVIL_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) _TYP_PATH = os.path.abspath(os.path.join(_CATAPULT_PATH, 'third_party', 'typ')) sys.path.append(_TYP_PATH) @@ -19,7 +19,24 @@ sys.path.append(_DEVIL_PATH) from devil.android import device_test_case +def _SetUpLogging(): + parsed_args = typ.arg_parser.ArgumentParser().parse_args(args=sys.argv[1:]) + verbosity = parsed_args.verbose + level = None + if verbosity == 0: + level = logging.WARNING + elif verbosity == 1: + level = logging.INFO + elif verbosity >= 2: + level = logging.DEBUG + else: + raise RuntimeError( + 'Logging verbosity of {} is not allowed.'.format(verbosity)) + logging.basicConfig(level=level) + + def main(): + _SetUpLogging() runner = typ.Runner() runner.setup_fn = device_test_case.PrepareDevices return runner.main( @@ -28,5 +45,6 @@ def main(): suffixes=['*_devicetest.py'], top_level_dir=_DEVIL_PATH) + if __name__ == '__main__': sys.exit(main()) diff --git a/catapult/devil/bin/run_py_tests b/catapult/devil/bin/run_py_tests index a74fa838..112451c6 100755 --- a/catapult/devil/bin/run_py_tests +++ b/catapult/devil/bin/run_py_tests @@ -6,10 +6,9 @@ import os import sys -_CATAPULT_PATH = os.path.abspath(os.path.join( - os.path.dirname(__file__), '..', '..')) -_DEVIL_PATH = os.path.abspath(os.path.join( - os.path.dirname(__file__), '..')) +_CATAPULT_PATH = os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..')) +_DEVIL_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) sys.path.append(_CATAPULT_PATH) from catapult_build import run_with_typ @@ -24,5 +23,6 @@ def main(): return run_with_typ.Run(top_level_dir=_DEVIL_PATH) + if __name__ == '__main__': sys.exit(main()) diff --git a/catapult/devil/build/cipd.yaml b/catapult/devil/build/cipd.yaml new file mode 100644 index 00000000..fcb4db3b --- /dev/null +++ b/catapult/devil/build/cipd.yaml @@ -0,0 +1,14 @@ +package: chromium/third_party/catapult/devil/${platform} + +description: All of devil along with its dependencies in catapult. + +platforms: + - linux-amd64 + +root: ../../ + +data: + - dir: common/py_utils + - dir: dependency_manager + - dir: devil + - dir: third_party/zipfile diff --git a/catapult/devil/devil/android/apk_helper.py b/catapult/devil/devil/android/apk_helper.py index abdf9071..fdece072 100644 --- a/catapult/devil/devil/android/apk_helper.py +++ b/catapult/devil/devil/android/apk_helper.py @@ -1,42 +1,97 @@ # Copyright (c) 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Module containing utilities for apk packages.""" +import contextlib +import logging +import os import re -import xml.etree.ElementTree +import shutil +import tempfile import zipfile from devil import base_error from devil.android.ndk import abis from devil.android.sdk import aapt +from devil.android.sdk import bundletool +from devil.android.sdk import split_select from devil.utils import cmd_helper +_logger = logging.getLogger(__name__) -_MANIFEST_ATTRIBUTE_RE = re.compile( - r'\s*A: ([^\(\)= ]*)(?:\([^\(\)= ]*\))?=' - r'(?:"(.*)" \(Raw: .*\)|\(type.*?\)(.*))$') +_MANIFEST_ATTRIBUTE_RE = re.compile(r'\s*A: ([^\(\)= ]*)(?:\([^\(\)= ]*\))?=' + r'(?:"(.*)" \(Raw: .*\)|\(type.*?\)(.*))$') _MANIFEST_ELEMENT_RE = re.compile(r'\s*(?:E|N): (\S*) .*$') +_BASE_APK_APKS_RE = re.compile(r'^splits/base-master.*\.apk$') + + +class ApkHelperError(base_error.BaseError): + """Exception for APK helper failures.""" + + def __init__(self, message): + super(ApkHelperError, self).__init__(message) + + +@contextlib.contextmanager +def _DeleteHelper(files, to_delete): + """Context manager that returns |files| and deletes |to_delete| on exit.""" + try: + yield files + finally: + paths = to_delete if isinstance(to_delete, list) else [to_delete] + for path in paths: + if os.path.isfile(path): + os.remove(path) + elif os.path.isdir(path): + shutil.rmtree(path) + else: + raise ApkHelperError('Cannot delete %s' % path) + + +@contextlib.contextmanager +def _NoopFileHelper(files): + """Context manager that returns |files|.""" + yield files def GetPackageName(apk_path): """Returns the package name of the apk.""" - return ApkHelper(apk_path).GetPackageName() + return ToHelper(apk_path).GetPackageName() # TODO(jbudorick): Deprecate and remove this function once callers have been # converted to ApkHelper.GetInstrumentationName def GetInstrumentationName(apk_path): """Returns the name of the Instrumentation in the apk.""" - return ApkHelper(apk_path).GetInstrumentationName() + return ToHelper(apk_path).GetInstrumentationName() def ToHelper(path_or_helper): """Creates an ApkHelper unless one is already given.""" - if isinstance(path_or_helper, basestring): + if not isinstance(path_or_helper, basestring): + return path_or_helper + elif path_or_helper.endswith('.apk'): return ApkHelper(path_or_helper) - return path_or_helper + elif path_or_helper.endswith('.apks'): + return ApksHelper(path_or_helper) + elif path_or_helper.endswith('_bundle'): + return BundleScriptHelper(path_or_helper) + + raise ApkHelperError('Unrecognized APK format %s' % path_or_helper) + + +def ToSplitHelper(path_or_helper, split_apks): + if isinstance(path_or_helper, SplitApkHelper): + if sorted(path_or_helper.split_apk_paths) != sorted(split_apks): + raise ApkHelperError('Helper has different split APKs') + return path_or_helper + elif (isinstance(path_or_helper, basestring) + and path_or_helper.endswith('.apk')): + return SplitApkHelper(path_or_helper, split_apks) + + raise ApkHelperError( + 'Unrecognized APK format %s, %s' % (path_or_helper, split_apks)) # To parse the manifest, the function uses a node stack where at each level of @@ -48,8 +103,8 @@ def ToHelper(path_or_helper): # matches the height of the stack). Each line parsed (either an attribute or an # element) is added to the node at the top of the stack (after the stack has # been popped/pushed due to indentation). -def _ParseManifestFromApk(apk): - aapt_output = aapt.Dump('xmltree', apk.path, 'AndroidManifest.xml') +def _ParseManifestFromApk(apk_path): + aapt_output = aapt.Dump('xmltree', apk_path, 'AndroidManifest.xml') parsed_manifest = {} node_stack = [parsed_manifest] indent = ' ' @@ -69,7 +124,8 @@ def _ParseManifestFromApk(apk): # If namespaces are stripped, aapt still outputs the full url to the # namespace and appends it to the attribute names. - line = line.replace('http://schemas.android.com/apk/res/android:', 'android:') + line = line.replace('http://schemas.android.com/apk/res/android:', + 'android:') indent_depth = 0 while line[(len(indent) * indent_depth):].startswith(indent): @@ -97,9 +153,9 @@ def _ParseManifestFromApk(apk): if m: manifest_key = m.group(1) if manifest_key in node: - raise base_error.BaseError( - "A single attribute should have one key and one value: {}" - .format(line)) + raise ApkHelperError( + "A single attribute should have one key and one value: {}".format( + line)) else: node[manifest_key] = m.group(2) or m.group(3) continue @@ -107,47 +163,6 @@ def _ParseManifestFromApk(apk): return parsed_manifest -def _ParseManifestFromBundle(bundle): - cmd = [bundle.path, 'dump-manifest'] - status, stdout, stderr = cmd_helper.GetCmdStatusOutputAndError(cmd) - if status != 0: - raise Exception('Failed running {} with output\n{}\n{}'.format( - ' '.join(cmd), stdout, stderr)) - return ParseManifestFromXml(stdout) - - -def ParseManifestFromXml(xml_str): - """Parse an android bundle manifest. - - As ParseManifestFromAapt, but uses the xml output from bundletool. Each - element is a dict, mapping attribute or children by name. Attributes map to - a dict (as they are unique), children map to a list of dicts (as there may - be multiple children with the same name). - - Args: - xml_str (str) An xml string that is an android manifest. - - Returns: - A dict holding the parsed manifest, as with ParseManifestFromAapt. - """ - root = xml.etree.ElementTree.fromstring(xml_str) - return {root.tag: [_ParseManifestXMLNode(root)]} - - -def _ParseManifestXMLNode(node): - out = {} - for name, value in node.attrib.items(): - cleaned_name = name.replace( - '{http://schemas.android.com/apk/res/android}', - 'android:').replace( - '{http://schemas.android.com/tools}', - 'tools:') - out[cleaned_name] = value - for child in node: - out.setdefault(child.tag, []).append(_ParseManifestXMLNode(child)) - return out - - def _ParseNumericKey(obj, key, default=0): val = obj.get(key) if val is None: @@ -155,6 +170,13 @@ def _ParseNumericKey(obj, key, default=0): return int(val, 0) +def _SplitLocaleString(locale): + split_locale = locale.split('-') + if len(split_locale) != 2: + raise ApkHelperError('Locale has incorrect format: {}'.format(locale)) + return tuple(split_locale) + + class _ExportedActivity(object): def __init__(self, name): self.name = name @@ -186,26 +208,32 @@ def _IterateExportedActivities(manifest_info): yield activity -class ApkHelper(object): +class BaseApkHelper(object): + """Abstract base class representing an installable Android app.""" - def __init__(self, path): - self._apk_path = path + def __init__(self): self._manifest = None @property def path(self): - return self._apk_path + raise NotImplementedError() - @property - def is_bundle(self): - return self._apk_path.endswith('_bundle') + def __repr__(self): + return '%s(%s)' % (self.__class__.__name__, self.path) + + def _GetBaseApkPath(self): + """Returns context manager providing path to this app's base APK. + + Must be implemented by subclasses. + """ + raise NotImplementedError() def GetActivityName(self): """Returns the name of the first launcher Activity in the apk.""" manifest_info = self._GetManifest() for activity in _IterateExportedActivities(manifest_info): - if ('android.intent.action.MAIN' in activity.actions and - 'android.intent.category.LAUNCHER' in activity.categories): + if ('android.intent.action.MAIN' in activity.actions + and 'android.intent.category.LAUNCHER' in activity.categories): return self._ResolveName(activity.name) return None @@ -213,23 +241,23 @@ class ApkHelper(object): """Returns name of the first action=View Activity that can handle http.""" manifest_info = self._GetManifest() for activity in _IterateExportedActivities(manifest_info): - if ('android.intent.action.VIEW' in activity.actions and - 'http' in activity.schemes): + if ('android.intent.action.VIEW' in activity.actions + and 'http' in activity.schemes): return self._ResolveName(activity.name) return None - def GetInstrumentationName( - self, default='android.test.InstrumentationTestRunner'): + def GetInstrumentationName(self, + default='android.test.InstrumentationTestRunner'): """Returns the name of the Instrumentation in the apk.""" all_instrumentations = self.GetAllInstrumentations(default=default) if len(all_instrumentations) != 1: - raise base_error.BaseError( + raise ApkHelperError( 'There is more than one instrumentation. Expected one.') else: return self._ResolveName(all_instrumentations[0]['android:name']) - def GetAllInstrumentations( - self, default='android.test.InstrumentationTestRunner'): + def GetAllInstrumentations(self, + default='android.test.InstrumentationTestRunner'): """Returns a list of all Instrumentations in the apk.""" try: return self._GetManifest()['manifest'][0]['instrumentation'] @@ -242,13 +270,15 @@ class ApkHelper(object): try: return manifest_info['manifest'][0]['package'] except KeyError: - raise Exception('Failed to determine package name of %s' % self._apk_path) + raise ApkHelperError('Failed to determine package name of %s' % self.path) def GetPermissions(self): manifest_info = self._GetManifest() try: - return [p['android:name'] for - p in manifest_info['manifest'][0]['uses-permission']] + return [ + p['android:name'] + for p in manifest_info['manifest'][0]['uses-permission'] + ] except KeyError: return [] @@ -323,7 +353,9 @@ class ApkHelper(object): def GetTargetSdkVersion(self): """Returns the targetSdkVersion as a string, or None if not available. - Note: this cannot always be cast to an integer.""" + Note: this cannot always be cast to an integer. If this application targets + a pre-release SDK, this returns the SDK codename instead (ex. "R"). + """ manifest_info = self._GetManifest() try: uses_sdk = manifest_info['manifest'][0]['uses-sdk'][0] @@ -343,11 +375,8 @@ class ApkHelper(object): def _GetManifest(self): if not self._manifest: - app = ToHelper(self._apk_path) - if app.is_bundle: - self._manifest = _ParseManifestFromBundle(app) - else: - self._manifest = _ParseManifestFromApk(app) + with self._GetBaseApkPath() as base_apk_path: + self._manifest = _ParseManifestFromApk(base_apk_path) return self._manifest def _ResolveName(self, name): @@ -357,8 +386,9 @@ class ApkHelper(object): return name def _ListApkPaths(self): - with zipfile.ZipFile(self._apk_path) as z: - return z.namelist() + with self._GetBaseApkPath() as base_apk_path: + with zipfile.ZipFile(base_apk_path) as z: + return z.namelist() def GetAbis(self): """Returns a list of ABIs in the apk (empty list if no native code).""" @@ -381,4 +411,197 @@ class ApkHelper(object): output.add(abi) return sorted(output) except KeyError: - raise base_error.BaseError('Unexpected ABI in lib/* folder.') + raise ApkHelperError('Unexpected ABI in lib/* folder.') + + def GetApkPaths(self, + device, + modules=None, + allow_cached_props=False, + additional_locales=None): + """Returns context manager providing list of split APK paths for |device|. + + The paths may be deleted when the context manager exits. Must be implemented + by subclasses. + + args: + device: The device for which to return split APKs. + modules: Extra feature modules to install. + allow_cached_props: Allow using cache when querying propery values from + |device|. + """ + # pylint: disable=unused-argument + raise NotImplementedError() + + @staticmethod + def SupportsSplits(): + return False + + +class ApkHelper(BaseApkHelper): + """Represents a single APK Android app.""" + + def __init__(self, apk_path): + super(ApkHelper, self).__init__() + self._apk_path = apk_path + + @property + def path(self): + return self._apk_path + + def _GetBaseApkPath(self): + return _NoopFileHelper(self._apk_path) + + def GetApkPaths(self, + device, + modules=None, + allow_cached_props=False, + additional_locales=None): + if modules: + raise ApkHelperError('Cannot install modules when installing single APK') + return _NoopFileHelper([self._apk_path]) + + +class SplitApkHelper(BaseApkHelper): + """Represents a multi APK Android app.""" + + def __init__(self, base_apk_path, split_apk_paths): + super(SplitApkHelper, self).__init__() + self._base_apk_path = base_apk_path + self._split_apk_paths = split_apk_paths + + @property + def path(self): + return self._base_apk_path + + @property + def split_apk_paths(self): + return self._split_apk_paths + + def __repr__(self): + return '%s(%s, %s)' % (self.__class__.__name__, self.path, + self.split_apk_paths) + + def _GetBaseApkPath(self): + return _NoopFileHelper(self._base_apk_path) + + def GetApkPaths(self, + device, + modules=None, + allow_cached_props=False, + additional_locales=None): + if modules: + raise ApkHelperError('Cannot install modules when installing single APK') + splits = split_select.SelectSplits( + device, + self.path, + self.split_apk_paths, + allow_cached_props=allow_cached_props) + if len(splits) == 1: + _logger.warning('split-select did not select any from %s', splits) + return _NoopFileHelper([self._base_apk_path] + splits) + + #override + @staticmethod + def SupportsSplits(): + return True + + +class BaseBundleHelper(BaseApkHelper): + """Abstract base class representing an Android app bundle.""" + + def _GetApksPath(self): + """Returns context manager providing path to the bundle's APKS archive. + + Must be implemented by subclasses. + """ + raise NotImplementedError() + + def _GetBaseApkPath(self): + try: + base_apk_path = tempfile.mkdtemp() + with self._GetApksPath() as apks_path: + with zipfile.ZipFile(apks_path) as z: + base_apks = [s for s in z.namelist() if _BASE_APK_APKS_RE.match(s)] + if len(base_apks) < 1: + raise ApkHelperError('Cannot find base APK in %s' % self.path) + z.extract(base_apks[0], base_apk_path) + return _DeleteHelper( + os.path.join(base_apk_path, base_apks[0]), base_apk_path) + except: + shutil.rmtree(base_apk_path) + raise + + def GetApkPaths(self, + device, + modules=None, + allow_cached_props=False, + additional_locales=None): + locales = [device.GetLocale()] + if additional_locales: + locales.extend(_SplitLocaleString(l) for l in additional_locales) + with self._GetApksPath() as apks_path: + try: + split_dir = tempfile.mkdtemp() + # TODO(tiborg): Support all locales. + bundletool.ExtractApks(split_dir, apks_path, + device.product_cpu_abis, locales, + device.GetFeatures(), device.pixel_density, + device.build_version_sdk, modules) + splits = [os.path.join(split_dir, p) for p in os.listdir(split_dir)] + return _DeleteHelper(splits, split_dir) + except: + shutil.rmtree(split_dir) + raise + + #override + @staticmethod + def SupportsSplits(): + return True + + +class ApksHelper(BaseBundleHelper): + """Represents a bundle's APKS archive.""" + + def __init__(self, apks_path): + super(ApksHelper, self).__init__() + self._apks_path = apks_path + + @property + def path(self): + return self._apks_path + + def _GetApksPath(self): + return _NoopFileHelper(self._apks_path) + + +class BundleScriptHelper(BaseBundleHelper): + """Represents a bundle install script.""" + + def __init__(self, bundle_script_path): + super(BundleScriptHelper, self).__init__() + self._bundle_script_path = bundle_script_path + + @property + def path(self): + return self._bundle_script_path + + def _GetApksPath(self): + apks_path = None + try: + fd, apks_path = tempfile.mkstemp(suffix='.apks') + os.close(fd) + cmd = [ + self._bundle_script_path, + 'build-bundle-apks', + '--output-apks', + apks_path, + ] + status, stdout, stderr = cmd_helper.GetCmdStatusOutputAndError(cmd) + if status != 0: + raise ApkHelperError('Failed running {} with output\n{}\n{}'.format( + ' '.join(cmd), stdout, stderr)) + return _DeleteHelper(apks_path, apks_path) + except: + if apks_path: + os.remove(apks_path) + raise diff --git a/catapult/devil/devil/android/apk_helper_test.py b/catapult/devil/devil/android/apk_helper_test.py index 3258bb01..6ac7fde2 100755 --- a/catapult/devil/devil/android/apk_helper_test.py +++ b/catapult/devil/devil/android/apk_helper_test.py @@ -7,7 +7,6 @@ import collections import os import unittest -from devil import base_error from devil import devil_env from devil.android import apk_helper from devil.android.ndk import abis @@ -16,7 +15,6 @@ from devil.utils import mock_calls with devil_env.SysPath(devil_env.PYMOCK_PATH): import mock # pylint: disable=import-error - # pylint: disable=line-too-long _MANIFEST_DUMP = """N: android=http://schemas.android.com/apk/res/android E: manifest (line=1) @@ -135,6 +133,7 @@ _NO_NAMESPACE_MANIFEST_DUMP = """E: manifest (line=1) A: http://schemas.android.com/apk/res/android:name(0x01010003)="org.chromium.RandomTestRunner" (Raw: "org.chromium.RandomTestRunner") A: http://schemas.android.com/apk/res/android:targetPackage(0x01010021)="org.chromium.random_package" (Raw:"org.chromium.random_pacakge") """ + # pylint: enable=line-too-long @@ -143,30 +142,70 @@ def _MockAaptDump(manifest_dump): 'devil.android.sdk.aapt.Dump', mock.Mock(side_effect=None, return_value=manifest_dump.split('\n'))) + def _MockListApkPaths(files): - return mock.patch( - 'devil.android.apk_helper.ApkHelper._ListApkPaths', - mock.Mock(side_effect=None, return_value=files)) + return mock.patch('devil.android.apk_helper.ApkHelper._ListApkPaths', + mock.Mock(side_effect=None, return_value=files)) + + +class _MockDeviceUtils(object): + def __init__(self): + self.product_cpu_abi = abis.ARM_64 + self.product_cpu_abis = [abis.ARM_64, abis.ARM] + self.pixel_density = 500 + self.build_version_sdk = 28 + + def GetLocale(self): + # pylint: disable=no-self-use + return ('en', 'US') + + def GetFeatures(self): + # pylint: disable=no-self-use + return [ + 'android.hardware.wifi', + 'android.hardware.nfc', + ] + class ApkHelperTest(mock_calls.TestCase): + def testToHelperApk(self): + apk = apk_helper.ToHelper('abc.apk') + self.assertTrue(isinstance(apk, apk_helper.ApkHelper)) + + def testToHelperApks(self): + apk = apk_helper.ToHelper('abc.apks') + self.assertTrue(isinstance(apk, apk_helper.ApksHelper)) + + def testToHelperBundleScript(self): + apk = apk_helper.ToHelper('abc_bundle') + self.assertTrue(isinstance(apk, apk_helper.BundleScriptHelper)) + + def testToHelperSplitApk(self): + apk = apk_helper.ToSplitHelper('abc.apk', ['a.apk', 'b.apk']) + self.assertTrue(isinstance(apk, apk_helper.SplitApkHelper)) + + def testToHelperSplitException(self): + with self.assertRaises(apk_helper.ApkHelperError): + apk_helper.ToSplitHelper( + apk_helper.ToHelper('abc.apk'), ['a.apk', 'b.apk']) def testGetInstrumentationName(self): with _MockAaptDump(_MANIFEST_DUMP): helper = apk_helper.ApkHelper('') - with self.assertRaises(base_error.BaseError): + with self.assertRaises(apk_helper.ApkHelperError): helper.GetInstrumentationName() def testGetActivityName(self): with _MockAaptDump(_MANIFEST_DUMP): helper = apk_helper.ApkHelper('') - self.assertEquals( - helper.GetActivityName(), 'org.chromium.abc.MainActivity') + self.assertEquals(helper.GetActivityName(), + 'org.chromium.abc.MainActivity') def testGetViewActivityName(self): with _MockAaptDump(_MANIFEST_DUMP): helper = apk_helper.ApkHelper('') - self.assertEquals( - helper.GetViewActivityName(), 'org.chromium.ViewActivity') + self.assertEquals(helper.GetViewActivityName(), + 'org.chromium.ViewActivity') def testGetAllInstrumentations(self): with _MockAaptDump(_MANIFEST_DUMP): @@ -275,107 +314,131 @@ class ApkHelperTest(mock_calls.TestCase): def testGetArchitectures(self): AbiPair = collections.namedtuple('AbiPair', ['abi32bit', 'abi64bit']) - for abi_pair in [AbiPair('lib/' + abis.ARM, 'lib/' + abis.ARM_64), - AbiPair('lib/' + abis.X86, 'lib/' + abis.X86_64)]: + for abi_pair in [ + AbiPair('lib/' + abis.ARM, 'lib/' + abis.ARM_64), + AbiPair('lib/' + abis.X86, 'lib/' + abis.X86_64) + ]: with _MockListApkPaths([abi_pair.abi32bit]): helper = apk_helper.ApkHelper('') - self.assertEquals(set([os.path.basename(abi_pair.abi32bit), - os.path.basename(abi_pair.abi64bit)]), - set(helper.GetAbis())) + self.assertEquals( + set([ + os.path.basename(abi_pair.abi32bit), + os.path.basename(abi_pair.abi64bit) + ]), set(helper.GetAbis())) with _MockListApkPaths([abi_pair.abi32bit, abi_pair.abi64bit]): helper = apk_helper.ApkHelper('') - self.assertEquals(set([os.path.basename(abi_pair.abi32bit), - os.path.basename(abi_pair.abi64bit)]), - set(helper.GetAbis())) + self.assertEquals( + set([ + os.path.basename(abi_pair.abi32bit), + os.path.basename(abi_pair.abi64bit) + ]), set(helper.GetAbis())) with _MockListApkPaths([abi_pair.abi64bit]): helper = apk_helper.ApkHelper('') - self.assertEquals(set([os.path.basename(abi_pair.abi64bit)]), - set(helper.GetAbis())) - - def testParseXmlManifest(self): - self.assertEquals({ - 'manifest': [ - {'android:compileSdkVersion': '28', - 'android:versionCode': '2', - 'uses-sdk': [ - {'android:minSdkVersion': '24', - 'android:targetSdkVersion': '28'}], - 'uses-permission': [ - {'android:name': - 'android.permission.ACCESS_COARSE_LOCATION'}, - {'android:name': - 'android.permission.ACCESS_NETWORK_STATE'}], - 'application': [ - {'android:allowBackup': 'true', - 'android:extractNativeLibs': 'false', - 'android:fullBackupOnly': 'false', - 'meta-data': [ - {'android:name': 'android.allow_multiple', - 'android:value': 'true'}, - {'android:name': 'multiwindow', - 'android:value': 'true'}], - 'activity': [ - {'android:configChanges': '0x00001fb3', - 'android:excludeFromRecents': 'true', - 'android:name': 'ChromeLauncherActivity', - 'intent-filter': [ - {'action': [ - {'android:name': 'dummy.action'}], - 'category': [ - {'android:name': 'DAYDREAM'}, - {'android:name': 'CARDBOARD'}]}]}, - {'android:enabled': 'false', - 'android:name': 'MediaLauncherActivity', - 'intent-filter': [ - {'tools:ignore': 'AppLinkUrlError', - 'action': [{'android:name': 'VIEW'}], - 'category': [{'android:name': 'DEFAULT'}], - 'data': [ - {'android:mimeType': 'audio/*'}, - {'android:mimeType': 'image/*'}, - {'android:mimeType': 'video/*'}, - {'android:scheme': 'file'}, - {'android:scheme': 'content'}]}]}]}]}]}, - apk_helper.ParseManifestFromXml(""" - <manifest xmlns:android="http://schemas.android.com/apk/res/android" - xmlns:tools="http://schemas.android.com/tools" - android:compileSdkVersion="28" android:versionCode="2"> - <uses-sdk android:minSdkVersion="24" android:targetSdkVersion="28"/> - <uses-permission - android:name="android.permission.ACCESS_COARSE_LOCATION"/> - <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/> - <application android:allowBackup="true" - android:extractNativeLibs="false" - android:fullBackupOnly="false"> - <meta-data android:name="android.allow_multiple" - android:value="true"/> - <meta-data android:name="multiwindow" - android:value="true"/> - <activity android:configChanges="0x00001fb3" - android:excludeFromRecents="true" - android:name="ChromeLauncherActivity"> - <intent-filter> - <action android:name="dummy.action"/> - <category android:name="DAYDREAM"/> - <category android:name="CARDBOARD"/> - </intent-filter> - </activity> - <activity android:enabled="false" - android:name="MediaLauncherActivity"> - <intent-filter tools:ignore="AppLinkUrlError"> - <action android:name="VIEW"/> - - <category android:name="DEFAULT"/> - - <data android:mimeType="audio/*"/> - <data android:mimeType="image/*"/> - <data android:mimeType="video/*"/> - <data android:scheme="file"/> - <data android:scheme="content"/> - </intent-filter> - </activity> - </application> - </manifest>""")) + self.assertEquals( + set([os.path.basename(abi_pair.abi64bit)]), set(helper.GetAbis())) + + def testGetSplitsApk(self): + apk = apk_helper.ToHelper('abc.apk') + with apk.GetApkPaths(_MockDeviceUtils()) as apk_paths: + self.assertEquals(apk_paths, ['abc.apk']) + + def testGetSplitsApkModulesException(self): + apk = apk_helper.ToHelper('abc.apk') + with self.assertRaises(apk_helper.ApkHelperError): + apk.GetApkPaths(None, modules=['a']) + + def testGetSplitsApks(self): + apk = apk_helper.ToHelper('abc.apks') + with self.assertCalls( + (mock.call.tempfile.mkdtemp(), + '/tmp'), + (mock.call.devil.android.sdk.bundletool.ExtractApks( + '/tmp', 'abc.apks', ['arm64-v8a', 'armeabi-v7a'], [('en', 'US')], + ['android.hardware.wifi', 'android.hardware.nfc'], 500, 28, None)), + (mock.call.os.listdir('/tmp'), ['base-master.apk', 'foo-master.apk']), + (mock.call.shutil.rmtree('/tmp')), + ),\ + apk.GetApkPaths(_MockDeviceUtils()) as apk_paths: + self.assertEquals(apk_paths, + ['/tmp/base-master.apk', '/tmp/foo-master.apk']) + + def testGetSplitsApksWithModules(self): + apk = apk_helper.ToHelper('abc.apks') + with self.assertCalls( + (mock.call.tempfile.mkdtemp(), + '/tmp'), + (mock.call.devil.android.sdk.bundletool.ExtractApks( + '/tmp', 'abc.apks', ['arm64-v8a', 'armeabi-v7a'], [('en', 'US')], + ['android.hardware.wifi', 'android.hardware.nfc'], 500, 28, + ['bar'])), + (mock.call.os.listdir('/tmp'), + ['base-master.apk', 'foo-master.apk', 'bar-master.apk']), + (mock.call.shutil.rmtree('/tmp')), + ),\ + apk.GetApkPaths(_MockDeviceUtils(), ['bar']) as apk_paths: + self.assertEquals(apk_paths, [ + '/tmp/base-master.apk', '/tmp/foo-master.apk', '/tmp/bar-master.apk' + ]) + + def testGetSplitsApksWithAdditionalLocales(self): + apk = apk_helper.ToHelper('abc.apks') + with self.assertCalls( + (mock.call.tempfile.mkdtemp(), + '/tmp'), + (mock.call.devil.android.sdk.bundletool.ExtractApks( + '/tmp', 'abc.apks', ['arm64-v8a', 'armeabi-v7a'], + [('en', 'US'), ('es', 'ES'), ('fr', 'CA')], + ['android.hardware.wifi', 'android.hardware.nfc'], 500, 28, None)), + (mock.call.os.listdir('/tmp'), + ['base-master.apk', 'base-es.apk', 'base-fr.apk']), + (mock.call.shutil.rmtree('/tmp')), + ),\ + apk.GetApkPaths(_MockDeviceUtils(), + additional_locales=['es-ES', 'fr-CA']) as apk_paths: + self.assertEquals( + apk_paths, + ['/tmp/base-master.apk', '/tmp/base-es.apk', '/tmp/base-fr.apk']) + + def testGetSplitsApksWithAdditionalLocalesIncorrectFormat(self): + apk = apk_helper.ToHelper('abc.apks') + with self.assertRaises(apk_helper.ApkHelperError): + apk.GetApkPaths(_MockDeviceUtils(), additional_locales=['es']) + + def testGetSplitsSplitApk(self): + apk = apk_helper.ToSplitHelper('base.apk', + ['split1.apk', 'split2.apk', 'split3.apk']) + device = _MockDeviceUtils() + with self.assertCalls( + (mock.call.devil.android.sdk.split_select.SelectSplits( + device, + 'base.apk', ['split1.apk', 'split2.apk', 'split3.apk'], + allow_cached_props=False), ['split2.apk'])),\ + apk.GetApkPaths(device) as apk_paths: + self.assertEquals(apk_paths, ['base.apk', 'split2.apk']) + + def testGetSplitsBundleScript(self): + apk = apk_helper.ToHelper('abc_bundle') + device = _MockDeviceUtils() + with self.assertCalls( + (mock.call.tempfile.mkstemp(suffix='.apks'), (0, '/tmp/abc.apks')), + (mock.call.devil.utils.cmd_helper.GetCmdStatusOutputAndError([ + 'abc_bundle', 'build-bundle-apks', '--output-apks', '/tmp/abc.apks' + ]), (0, '', '')), + (mock.call.tempfile.mkdtemp(), '/tmp2'), + (mock.call.devil.android.sdk.bundletool.ExtractApks( + '/tmp2', '/tmp/abc.apks', ['arm64-v8a', 'armeabi-v7a'], + [('en', 'US')], ['android.hardware.wifi', 'android.hardware.nfc'], + 500, 28, ['bar'])), + (mock.call.os.listdir('/tmp2'), ['base-master.apk', 'bar-master.apk']), + (mock.call.os.path.isfile('/tmp/abc.apks'), True), + (mock.call.os.remove('/tmp/abc.apks')), + (mock.call.os.path.isfile('/tmp2'), False), + (mock.call.os.path.isdir('/tmp2'), True), + (mock.call.shutil.rmtree('/tmp2')), + ),\ + apk.GetApkPaths(device, modules=['bar']) as apk_paths: + self.assertEquals(apk_paths, + ['/tmp2/base-master.apk', '/tmp2/bar-master.apk']) if __name__ == '__main__': diff --git a/catapult/devil/devil/android/app_ui.py b/catapult/devil/devil/android/app_ui.py index 2b04e8b8..399c2ee3 100644 --- a/catapult/devil/devil/android/app_ui.py +++ b/catapult/devil/devil/android/app_ui.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Provides functionality to interact with UI elements of an Android app.""" import collections @@ -24,7 +23,6 @@ _RE_BOUNDS = re.compile( class _UiNode(object): - def __init__(self, device, xml_node, package=None): """Object to interact with a UI node from an xml snapshot. @@ -148,13 +146,11 @@ class _UiNode(object): def _NodeMatcher(self, kwargs): # Auto-complete resource-id's using the package name if available. resource_id = kwargs.get('resource_id') - if (resource_id is not None - and self._package is not None + if (resource_id is not None and self._package is not None and ':id/' not in resource_id): kwargs['resource_id'] = '%s:id/%s' % (self._package, resource_id) - criteria = [(k.replace('_', '-'), v) - for k, v in kwargs.iteritems() + criteria = [(k.replace('_', '-'), v) for k, v in kwargs.iteritems() if v is not None] if not criteria: raise TypeError('At least one search criteria should be specified') @@ -198,7 +194,7 @@ class AppUi(object): """ with device_temp_file.DeviceTempFile(self._device.adb) as dtemp: self._device.RunShellCommand(['uiautomator', 'dump', dtemp.name], - check_return=True) + check_return=True) xml_node = element_tree.fromstring( self._device.ReadFile(dtemp.name, force_pull=True)) return _UiNode(self._device, xml_node, package=self._package) @@ -237,6 +233,7 @@ class AppUi(object): device_errors.CommandTimeoutError if the node is not found before the timeout. """ + def node_found(): return self.GetUiNode(**kwargs) diff --git a/catapult/devil/devil/android/app_ui_test.py b/catapult/devil/devil/android/app_ui_test.py index 34729851..938fd408 100644 --- a/catapult/devil/devil/android/app_ui_test.py +++ b/catapult/devil/devil/android/app_ui_test.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Unit tests for the app_ui module.""" import unittest @@ -15,7 +14,6 @@ from devil.utils import geometry with devil_env.SysPath(devil_env.PYMOCK_PATH): import mock # pylint: disable=import-error - MOCK_XML_LOADING = ''' <?xml version='1.0' encoding='UTF-8' standalone='yes' ?> <hierarchy rotation="0"> @@ -24,7 +22,6 @@ MOCK_XML_LOADING = ''' </hierarchy> '''.strip() - MOCK_XML_LOADED = ''' <?xml version='1.0' encoding='UTF-8' standalone='yes' ?> <hierarchy rotation="0"> @@ -56,7 +53,6 @@ MOCK_XML_LOADED = ''' class UiAppTest(unittest.TestCase): - def setUp(self): self.device = mock.Mock() self.device.pixel_density = 320 # Each dp pixel is 2 real pixels. @@ -69,12 +65,13 @@ class UiAppTest(unittest.TestCase): Each time the method is called it will return a UI node for each string given in |xml_docs|, or rise a time out error when the list is exhausted. """ + # pylint: disable=protected-access def get_mock_root_ui_node(value): if isinstance(value, Exception): raise value - return app_ui._UiNode( - self.device, element_tree.fromstring(value), self.app.package) + return app_ui._UiNode(self.device, element_tree.fromstring(value), + self.app.package) xml_docs.append(device_errors.CommandTimeoutError('Timed out!')) @@ -92,20 +89,22 @@ class UiAppTest(unittest.TestCase): def testFind_byText(self): node = self.app.GetUiNode(text='Primary') - self.assertNodeHasAttribs(node, { - 'text': 'Primary', - 'content-desc': None, - 'resource-id': 'com.example.app:id/actionbar_title', - }) + self.assertNodeHasAttribs( + node, { + 'text': 'Primary', + 'content-desc': None, + 'resource-id': 'com.example.app:id/actionbar_title', + }) self.assertEquals(node.bounds, geometry.Rectangle([121, 50], [1424, 178])) def testFind_byContentDesc(self): node = self.app.GetUiNode(content_desc='Social') - self.assertNodeHasAttribs(node, { - 'text': None, - 'content-desc': 'Social', - 'resource-id': 'com.example.app:id/image_view', - }) + self.assertNodeHasAttribs( + node, { + 'text': None, + 'content-desc': 'Social', + 'resource-id': 'com.example.app:id/image_view', + }) self.assertEquals(node.bounds, geometry.Rectangle([16, 466], [128, 578])) def testFind_byResourceId_autocompleted(self): @@ -123,12 +122,13 @@ class UiAppTest(unittest.TestCase): }) def testFind_byMultiple(self): - node = self.app.GetUiNode(resource_id='image_view', - content_desc='Promotions') - self.assertNodeHasAttribs(node, { - 'content-desc': 'Promotions', - 'resource-id': 'com.example.app:id/image_view', - }) + node = self.app.GetUiNode( + resource_id='image_view', content_desc='Promotions') + self.assertNodeHasAttribs( + node, { + 'content-desc': 'Promotions', + 'resource-id': 'com.example.app:id/image_view', + }) self.assertEquals(node.bounds, geometry.Rectangle([16, 578], [128, 690])) def testFind_notFound(self): @@ -142,8 +142,8 @@ class UiAppTest(unittest.TestCase): def testGetChildren(self): node = self.app.GetUiNode(resource_id='mini_drawer') - self.assertNodeHasAttribs( - node[0], {'resource-id': 'com.example.app:id/avatar'}) + self.assertNodeHasAttribs(node[0], + {'resource-id': 'com.example.app:id/avatar'}) self.assertNodeHasAttribs(node[1], {'content-desc': 'Primary'}) self.assertNodeHasAttribs(node[2], {'content-desc': 'Social'}) self.assertNodeHasAttribs(node[3], {'content-desc': 'Promotions'}) diff --git a/catapult/devil/devil/android/battery_utils.py b/catapult/devil/devil/android/battery_utils.py index c41c19a2..e8134d2b 100644 --- a/catapult/devil/devil/android/battery_utils.py +++ b/catapult/devil/devil/android/battery_utils.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Provides a variety of device interactions with power. """ # pylint: disable=unused-argument @@ -25,7 +24,7 @@ _DEFAULT_RETRIES = 3 _DEVICE_PROFILES = [ - { + { 'name': ['Nexus 4'], 'enable_command': ( 'echo 0 > /sys/module/pm8921_charger/parameters/disabled && ' @@ -36,8 +35,8 @@ _DEVICE_PROFILES = [ 'charge_counter': None, 'voltage': None, 'current': None, - }, - { + }, + { 'name': ['Nexus 5'], # Nexus 5 # Setting the HIZ bit of the bq24192 causes the charger to actually ignore @@ -56,8 +55,8 @@ _DEVICE_PROFILES = [ 'charge_counter': None, 'voltage': None, 'current': None, - }, - { + }, + { 'name': ['Nexus 6'], 'enable_command': ( 'echo 1 > /sys/class/power_supply/battery/charging_enabled && ' @@ -69,8 +68,8 @@ _DEVICE_PROFILES = [ '/sys/class/power_supply/max170xx_battery/charge_counter_ext'), 'voltage': '/sys/class/power_supply/max170xx_battery/voltage_now', 'current': '/sys/class/power_supply/max170xx_battery/current_now', - }, - { + }, + { 'name': ['Nexus 9'], 'enable_command': ( 'echo Disconnected > ' @@ -83,8 +82,8 @@ _DEVICE_PROFILES = [ 'charge_counter': '/sys/class/power_supply/battery/charge_counter_ext', 'voltage': '/sys/class/power_supply/battery/voltage_now', 'current': '/sys/class/power_supply/battery/current_now', - }, - { + }, + { 'name': ['Nexus 10'], 'enable_command': None, 'disable_command': None, @@ -92,8 +91,8 @@ _DEVICE_PROFILES = [ 'voltage': '/sys/class/power_supply/ds2784-fuelgauge/voltage_now', 'current': '/sys/class/power_supply/ds2784-fuelgauge/current_now', - }, - { + }, + { 'name': ['Nexus 5X'], 'enable_command': ( 'echo 1 > /sys/class/power_supply/battery/charging_enabled && ' @@ -104,8 +103,8 @@ _DEVICE_PROFILES = [ 'charge_counter': None, 'voltage': None, 'current': None, - }, - { # Galaxy s5 + }, + { # Galaxy s5 'name': ['SM-G900H'], 'enable_command': ( 'chmod 644 /sys/class/power_supply/battery/test_mode && ' @@ -122,8 +121,8 @@ _DEVICE_PROFILES = [ 'charge_counter': None, 'voltage': '/sys/class/power_supply/sec-fuelgauge/voltage_now', 'current': '/sys/class/power_supply/sec-charger/current_now', - }, - { # Galaxy s6, Galaxy s6, Galaxy s6 edge + }, + { # Galaxy s6, Galaxy s6, Galaxy s6 edge 'name': ['SM-G920F', 'SM-G920V', 'SM-G925V'], 'enable_command': ( 'chmod 644 /sys/class/power_supply/battery/test_mode && ' @@ -140,8 +139,8 @@ _DEVICE_PROFILES = [ 'charge_counter': None, 'voltage': '/sys/class/power_supply/max77843-fuelgauge/voltage_now', 'current': '/sys/class/power_supply/max77843-charger/current_now', - }, - { # Cherry Mobile One + }, + { # Cherry Mobile One 'name': ['W6210 (4560MMX_b fingerprint)'], 'enable_command': ( 'echo "0 0" > /proc/mtk_battery_cmd/current_cmd && ' @@ -152,7 +151,7 @@ _DEVICE_PROFILES = [ 'charge_counter': None, 'voltage': None, 'current': None, -}, + }, ] # The list of useful dumpsys columns. @@ -178,8 +177,9 @@ _MAX_CHARGE_ERROR = 20 class BatteryUtils(object): - - def __init__(self, device, default_timeout=_DEFAULT_TIMEOUT, + def __init__(self, + device, + default_timeout=_DEFAULT_TIMEOUT, default_retries=_DEFAULT_RETRIES): """BatteryUtils constructor. @@ -215,7 +215,7 @@ class BatteryUtils(object): """ self._DiscoverDeviceProfile() return (self._cache['profile']['enable_command'] != None - and self._cache['profile']['charge_counter'] != None) + and self._cache['profile']['charge_counter'] != None) @decorators.WithTimeoutAndRetriesFromInstance() def GetFuelGaugeChargeCounter(self, timeout=None, retries=None): @@ -235,10 +235,9 @@ class BatteryUtils(object): device_errors.CommandFailedError: If fuel gauge chip not found. """ if self.SupportsFuelGauge(): - return int(self._device.ReadFile( - self._cache['profile']['charge_counter'])) - raise device_errors.CommandFailedError( - 'Unable to find fuel gauge.') + return int( + self._device.ReadFile(self._cache['profile']['charge_counter'])) + raise device_errors.CommandFailedError('Unable to find fuel gauge.') @decorators.WithTimeoutAndRetriesFromInstance() def GetPowerData(self, timeout=None, retries=None): @@ -264,8 +263,7 @@ class BatteryUtils(object): if 'uids' not in self._cache: self._cache['uids'] = {} dumpsys_output = self._device.RunShellCommand( - ['dumpsys', 'batterystats', '-c'], - check_return=True, large_output=True) + ['dumpsys', 'batterystats', '-c'], check_return=True, large_output=True) csvreader = csv.reader(dumpsys_output) pwi_entries = collections.defaultdict(list) system_total = None @@ -273,32 +271,37 @@ class BatteryUtils(object): if entry[_DUMP_VERSION_INDEX] not in ['8', '9']: # Wrong dumpsys version. raise device_errors.DeviceVersionError( - 'Dumpsys version must be 8 or 9. "%s" found.' - % entry[_DUMP_VERSION_INDEX]) + 'Dumpsys version must be 8 or 9. "%s" found.' % + entry[_DUMP_VERSION_INDEX]) if _ROW_TYPE_INDEX < len(entry) and entry[_ROW_TYPE_INDEX] == 'uid': current_package = entry[_PACKAGE_NAME_INDEX] if (self._cache['uids'].get(current_package) - and self._cache['uids'].get(current_package) - != entry[_PACKAGE_UID_INDEX]): + and self._cache['uids'].get(current_package) != + entry[_PACKAGE_UID_INDEX]): raise device_errors.CommandFailedError( - 'Package %s found multiple times with different UIDs %s and %s' - % (current_package, self._cache['uids'][current_package], + 'Package %s found multiple times with different UIDs %s and %s' % + (current_package, self._cache['uids'][current_package], entry[_PACKAGE_UID_INDEX])) self._cache['uids'][current_package] = entry[_PACKAGE_UID_INDEX] elif (_PWI_POWER_CONSUMPTION_INDEX < len(entry) - and entry[_ROW_TYPE_INDEX] == 'pwi' - and entry[_PWI_AGGREGATION_INDEX] == 'l'): + and entry[_ROW_TYPE_INDEX] == 'pwi' + and entry[_PWI_AGGREGATION_INDEX] == 'l'): pwi_entries[entry[_PWI_UID_INDEX]].append( float(entry[_PWI_POWER_CONSUMPTION_INDEX])) elif (_PWS_POWER_CONSUMPTION_INDEX < len(entry) - and entry[_ROW_TYPE_INDEX] == 'pws' - and entry[_PWS_AGGREGATION_INDEX] == 'l'): + and entry[_ROW_TYPE_INDEX] == 'pws' + and entry[_PWS_AGGREGATION_INDEX] == 'l'): # This entry should only appear once. assert system_total is None system_total = float(entry[_PWS_POWER_CONSUMPTION_INDEX]) - per_package = {p: {'uid': uid, 'data': pwi_entries[uid]} - for p, uid in self._cache['uids'].iteritems()} + per_package = { + p: { + 'uid': uid, + 'data': pwi_entries[uid] + } + for p, uid in self._cache['uids'].iteritems() + } return {'system_total': system_total, 'per_package': per_package} @decorators.WithTimeoutAndRetriesFromInstance() @@ -314,8 +317,8 @@ class BatteryUtils(object): """ result = {} # Skip the first line, which is just a header. - for line in self._device.RunShellCommand( - ['dumpsys', 'battery'], check_return=True)[1:]: + for line in self._device.RunShellCommand(['dumpsys', 'battery'], + check_return=True)[1:]: # If usb charging has been disabled, an extra line of header exists. if 'UPDATES STOPPED' in line: logger.warning('Dumpsys battery not receiving updates. ' @@ -337,15 +340,16 @@ class BatteryUtils(object): Returns: True if the device is charging, false otherwise. """ + # Wrapper function so that we can use `RetryOnSystemCrash`. def GetBatteryInfoHelper(device): return self.GetBatteryInfo() - battery_info = crash_handler.RetryOnSystemCrash( - GetBatteryInfoHelper, self._device) + battery_info = crash_handler.RetryOnSystemCrash(GetBatteryInfoHelper, + self._device) for k in ('AC powered', 'USB powered', 'Wireless powered'): - if (k in battery_info and - battery_info[k].lower() in ('true', '1', 'yes')): + if (k in battery_info + and battery_info[k].lower() in ('true', '1', 'yes')): return True return False @@ -364,6 +368,7 @@ class BatteryUtils(object): reset power values. device_errors.DeviceVersionError: If device is not L or higher. """ + def battery_updates_disabled(): return self.GetCharging() is False @@ -386,6 +391,7 @@ class BatteryUtils(object): Raises: device_errors.DeviceVersionError: If device is not L or higher. """ + def battery_updates_enabled(): return (self.GetCharging() or not bool('UPDATES STOPPED' in self._device.RunShellCommand( @@ -437,15 +443,16 @@ class BatteryUtils(object): """ battery_level = int(self.GetBatteryInfo().get('level')) if not 0 < percent < 100: - raise ValueError('Discharge amount(%s) must be between 1 and 99' - % percent) + raise ValueError( + 'Discharge amount(%s) must be between 1 and 99' % percent) if battery_level is None: logger.warning('Unable to find current battery level. Cannot discharge.') return # Do not discharge if it would make battery level too low. if percent >= battery_level - 10: - logger.warning('Battery is too low or discharge amount requested is too ' - 'high. Cannot discharge phone %s percent.', percent) + logger.warning( + 'Battery is too low or discharge amount requested is too ' + 'high. Cannot discharge phone %s percent.', percent) return self._HardwareSetCharging(False) @@ -471,10 +478,8 @@ class BatteryUtils(object): device_errors.DeviceChargingError: If error while charging is detected. """ self.SetCharging(True) - charge_status = { - 'charge_failure_count': 0, - 'last_charge_value': 0 - } + charge_status = {'charge_failure_count': 0, 'last_charge_value': 0} + def device_charged(): battery_level = self.GetBatteryInfo().get('level') if battery_level is None: @@ -494,8 +499,8 @@ class BatteryUtils(object): if (not battery_level >= level and charge_status['charge_failure_count'] >= _MAX_CHARGE_ERROR): raise device_errors.DeviceChargingError( - 'Device not charging properly. Current level:%s Previous level:%s' - % (battery_level, charge_status['last_charge_value'])) + 'Device not charging properly. Current level:%s Previous level:%s' % + (battery_level, charge_status['last_charge_value'])) return battery_level >= level timeout_retry.WaitFor(device_charged, wait_period=wait_period) @@ -506,6 +511,7 @@ class BatteryUtils(object): temp: maximum temperature to allow in tenths of degrees c. wait_period: time in seconds to wait between checking. """ + def cool_device(): temp = self.GetBatteryInfo().get('temperature') if temp is None: @@ -554,7 +560,7 @@ class BatteryUtils(object): self._HardwareSetCharging(enabled) else: logger.info('Unable to disable charging via hardware. ' - 'Falling back to software disabling.') + 'Falling back to software disabling.') self.DisableBatteryUpdates() def _HardwareSetCharging(self, enabled, timeout=None, retries=None): @@ -575,8 +581,8 @@ class BatteryUtils(object): raise device_errors.CommandFailedError( 'Unable to find charging commands.') - command = (self._cache['profile']['enable_command'] if enabled - else self._cache['profile']['disable_command']) + command = (self._cache['profile']['enable_command'] + if enabled else self._cache['profile']['disable_command']) def verify_charging(): return self.GetCharging() == enabled @@ -629,17 +635,18 @@ class BatteryUtils(object): 'Cannot clear power data.') return False - self._device.RunShellCommand( - ['dumpsys', 'battery', 'set', 'usb', '1'], check_return=True) - self._device.RunShellCommand( - ['dumpsys', 'battery', 'set', 'ac', '1'], check_return=True) + self._device.RunShellCommand(['dumpsys', 'battery', 'set', 'usb', '1'], + check_return=True) + self._device.RunShellCommand(['dumpsys', 'battery', 'set', 'ac', '1'], + check_return=True) def test_if_clear(): - self._device.RunShellCommand( - ['dumpsys', 'batterystats', '--reset'], check_return=True) + self._device.RunShellCommand(['dumpsys', 'batterystats', '--reset'], + check_return=True) battery_data = self._device.RunShellCommand( ['dumpsys', 'batterystats', '--charged', '-c'], - check_return=True, large_output=True) + check_return=True, + large_output=True) for line in battery_data: l = line.split(',') if (len(l) > _PWI_POWER_CONSUMPTION_INDEX @@ -652,8 +659,8 @@ class BatteryUtils(object): timeout_retry.WaitFor(test_if_clear, wait_period=1) return True finally: - self._device.RunShellCommand( - ['dumpsys', 'battery', 'reset'], check_return=True) + self._device.RunShellCommand(['dumpsys', 'battery', 'reset'], + check_return=True) def _DiscoverDeviceProfile(self): """Checks and caches device information. diff --git a/catapult/devil/devil/android/battery_utils_test.py b/catapult/devil/devil/android/battery_utils_test.py index 07c74967..7bfb9551 100755 --- a/catapult/devil/devil/android/battery_utils_test.py +++ b/catapult/devil/devil/android/battery_utils_test.py @@ -2,7 +2,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Unit tests for the contents of battery_utils.py """ @@ -34,46 +33,64 @@ _DUMPSYS_OUTPUT = [ class BatteryUtilsTest(mock_calls.TestCase): _NEXUS_5 = { - 'name': 'Nexus 5', - 'witness_file': '/sys/kernel/debug/bq24192/INPUT_SRC_CONT', - 'enable_command': ( - 'echo 0x4A > /sys/kernel/debug/bq24192/INPUT_SRC_CONT && ' - 'echo 1 > /sys/class/power_supply/usb/online'), - 'disable_command': ( - 'echo 0xCA > /sys/kernel/debug/bq24192/INPUT_SRC_CONT && ' - 'chmod 644 /sys/class/power_supply/usb/online && ' - 'echo 0 > /sys/class/power_supply/usb/online'), - 'charge_counter': None, - 'voltage': None, - 'current': None, + 'name': + 'Nexus 5', + 'witness_file': + '/sys/kernel/debug/bq24192/INPUT_SRC_CONT', + 'enable_command': ( + 'echo 0x4A > /sys/kernel/debug/bq24192/INPUT_SRC_CONT && ' + 'echo 1 > /sys/class/power_supply/usb/online'), + 'disable_command': ( + 'echo 0xCA > /sys/kernel/debug/bq24192/INPUT_SRC_CONT && ' + 'chmod 644 /sys/class/power_supply/usb/online && ' + 'echo 0 > /sys/class/power_supply/usb/online'), + 'charge_counter': + None, + 'voltage': + None, + 'current': + None, } _NEXUS_6 = { - 'name': 'Nexus 6', - 'witness_file': None, - 'enable_command': None, - 'disable_command': None, - 'charge_counter': ( - '/sys/class/power_supply/max170xx_battery/charge_counter_ext'), - 'voltage': '/sys/class/power_supply/max170xx_battery/voltage_now', - 'current': '/sys/class/power_supply/max170xx_battery/current_now', + 'name': + 'Nexus 6', + 'witness_file': + None, + 'enable_command': + None, + 'disable_command': + None, + 'charge_counter': ( + '/sys/class/power_supply/max170xx_battery/charge_counter_ext'), + 'voltage': + '/sys/class/power_supply/max170xx_battery/voltage_now', + 'current': + '/sys/class/power_supply/max170xx_battery/current_now', } _NEXUS_10 = { - 'name': 'Nexus 10', - 'witness_file': None, - 'enable_command': None, - 'disable_command': None, - 'charge_counter': ( - '/sys/class/power_supply/ds2784-fuelgauge/charge_counter_ext'), - 'voltage': '/sys/class/power_supply/ds2784-fuelgauge/voltage_now', - 'current': '/sys/class/power_supply/ds2784-fuelgauge/current_now', + 'name': + 'Nexus 10', + 'witness_file': + None, + 'enable_command': + None, + 'disable_command': + None, + 'charge_counter': ( + '/sys/class/power_supply/ds2784-fuelgauge/charge_counter_ext'), + 'voltage': + '/sys/class/power_supply/ds2784-fuelgauge/voltage_now', + 'current': + '/sys/class/power_supply/ds2784-fuelgauge/current_now', } def ShellError(self, output=None, status=1): def action(cmd, *args, **kwargs): - raise device_errors.AdbShellCommandFailedError( - cmd, output, status, str(self.device)) + raise device_errors.AdbShellCommandFailedError(cmd, output, status, + str(self.device)) + if output is None: output = 'Permission denied\n' return action @@ -88,7 +105,6 @@ class BatteryUtilsTest(mock_calls.TestCase): class BatteryUtilsInitTest(unittest.TestCase): - def testInitWithDeviceUtil(self): serial = '0fedcba987654321' d = device_utils.DeviceUtils(serial) @@ -103,78 +119,75 @@ class BatteryUtilsInitTest(unittest.TestCase): class BatteryUtilsSetChargingTest(BatteryUtilsTest): - @mock.patch('time.sleep', mock.Mock()) def testHardwareSetCharging_enabled(self): self.battery._cache['profile'] = self._NEXUS_5 - with self.assertCalls( - (self.call.device.RunShellCommand( - mock.ANY, shell=True, check_return=True, as_root=True, - large_output=True), []), - (self.call.battery.GetCharging(), False), - (self.call.battery.GetCharging(), True)): + with self.assertCalls((self.call.device.RunShellCommand( + mock.ANY, + shell=True, + check_return=True, + as_root=True, + large_output=True), []), (self.call.battery.GetCharging(), False), + (self.call.battery.GetCharging(), True)): self.battery._HardwareSetCharging(True) def testHardwareSetCharging_alreadyEnabled(self): self.battery._cache['profile'] = self._NEXUS_5 - with self.assertCalls( - (self.call.device.RunShellCommand( - mock.ANY, shell=True, check_return=True, as_root=True, - large_output=True), []), - (self.call.battery.GetCharging(), True)): + with self.assertCalls((self.call.device.RunShellCommand( + mock.ANY, + shell=True, + check_return=True, + as_root=True, + large_output=True), []), (self.call.battery.GetCharging(), True)): self.battery._HardwareSetCharging(True) @mock.patch('time.sleep', mock.Mock()) def testHardwareSetCharging_disabled(self): self.battery._cache['profile'] = self._NEXUS_5 - with self.assertCalls( - (self.call.device.RunShellCommand( - mock.ANY, shell=True, check_return=True, as_root=True, - large_output=True), []), - (self.call.battery.GetCharging(), True), - (self.call.battery.GetCharging(), False)): + with self.assertCalls((self.call.device.RunShellCommand( + mock.ANY, + shell=True, + check_return=True, + as_root=True, + large_output=True), []), (self.call.battery.GetCharging(), True), + (self.call.battery.GetCharging(), False)): self.battery._HardwareSetCharging(False) class BatteryUtilsSetBatteryMeasurementTest(BatteryUtilsTest): - @mock.patch('time.sleep', mock.Mock()) def testBatteryMeasurementWifi(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=22): + with self.patch_call(self.call.device.build_version_sdk, return_value=22): with self.assertCalls( (self.call.battery._ClearPowerData(), True), (self.call.device.RunShellCommand( ['dumpsys', 'battery', 'set', 'ac', '0'], check_return=True), []), (self.call.device.RunShellCommand( ['dumpsys', 'battery', 'set', 'usb', '0'], check_return=True), - []), - (self.call.battery.GetCharging(), False), - (self.call.device.RunShellCommand( - ['dumpsys', 'battery', 'reset'], check_return=True), []), + []), (self.call.battery.GetCharging(), False), + (self.call.device.RunShellCommand(['dumpsys', 'battery', 'reset'], + check_return=True), []), (self.call.battery.GetCharging(), False), (self.call.device.RunShellCommand( ['dumpsys', 'battery'], check_return=True), ['UPDATES STOPPED']), (self.call.battery.GetCharging(), False), - (self.call.device.RunShellCommand( - ['dumpsys', 'battery'], check_return=True), [])): + (self.call.device.RunShellCommand(['dumpsys', 'battery'], + check_return=True), [])): with self.battery.BatteryMeasurement(): pass @mock.patch('time.sleep', mock.Mock()) def testBatteryMeasurementUsb(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=22): + with self.patch_call(self.call.device.build_version_sdk, return_value=22): with self.assertCalls( (self.call.battery._ClearPowerData(), True), (self.call.device.RunShellCommand( ['dumpsys', 'battery', 'set', 'ac', '0'], check_return=True), []), (self.call.device.RunShellCommand( ['dumpsys', 'battery', 'set', 'usb', '0'], check_return=True), - []), - (self.call.battery.GetCharging(), False), - (self.call.device.RunShellCommand( - ['dumpsys', 'battery', 'reset'], check_return=True), []), + []), (self.call.battery.GetCharging(), False), + (self.call.device.RunShellCommand(['dumpsys', 'battery', 'reset'], + check_return=True), []), (self.call.battery.GetCharging(), False), (self.call.device.RunShellCommand( ['dumpsys', 'battery'], check_return=True), ['UPDATES STOPPED']), @@ -184,86 +197,102 @@ class BatteryUtilsSetBatteryMeasurementTest(BatteryUtilsTest): class BatteryUtilsGetPowerData(BatteryUtilsTest): - def testGetPowerData(self): with self.assertCalls( - (self.call.device.RunShellCommand( - ['dumpsys', 'batterystats', '-c'], - check_return=True, large_output=True), - _DUMPSYS_OUTPUT)): + (self.call.device.RunShellCommand(['dumpsys', 'batterystats', '-c'], + check_return=True, + large_output=True), _DUMPSYS_OUTPUT)): data = self.battery.GetPowerData() check = { - 'system_total': 2000.0, - 'per_package': { - 'test_package1': {'uid': '1000', 'data': [1.0]}, - 'test_package2': {'uid': '1001', 'data': [2.0]} - } + 'system_total': 2000.0, + 'per_package': { + 'test_package1': { + 'uid': '1000', + 'data': [1.0] + }, + 'test_package2': { + 'uid': '1001', + 'data': [2.0] + } + } } self.assertEqual(data, check) def testGetPowerData_packageCollisionSame(self): self.battery._cache['uids'] = {'test_package1': '1000'} with self.assertCall( - self.call.device.RunShellCommand( - ['dumpsys', 'batterystats', '-c'], - check_return=True, large_output=True), - _DUMPSYS_OUTPUT): + self.call.device.RunShellCommand(['dumpsys', 'batterystats', '-c'], + check_return=True, + large_output=True), _DUMPSYS_OUTPUT): data = self.battery.GetPowerData() check = { - 'system_total': 2000.0, - 'per_package': { - 'test_package1': {'uid': '1000', 'data': [1.0]}, - 'test_package2': {'uid': '1001', 'data': [2.0]} - } + 'system_total': 2000.0, + 'per_package': { + 'test_package1': { + 'uid': '1000', + 'data': [1.0] + }, + 'test_package2': { + 'uid': '1001', + 'data': [2.0] + } + } } self.assertEqual(data, check) def testGetPowerData_packageCollisionDifferent(self): self.battery._cache['uids'] = {'test_package1': '1'} with self.assertCall( - self.call.device.RunShellCommand( - ['dumpsys', 'batterystats', '-c'], - check_return=True, large_output=True), - _DUMPSYS_OUTPUT): + self.call.device.RunShellCommand(['dumpsys', 'batterystats', '-c'], + check_return=True, + large_output=True), _DUMPSYS_OUTPUT): with self.assertRaises(device_errors.CommandFailedError): self.battery.GetPowerData() def testGetPowerData_cacheCleared(self): with self.assertCalls( - (self.call.device.RunShellCommand( - ['dumpsys', 'batterystats', '-c'], - check_return=True, large_output=True), - _DUMPSYS_OUTPUT)): + (self.call.device.RunShellCommand(['dumpsys', 'batterystats', '-c'], + check_return=True, + large_output=True), _DUMPSYS_OUTPUT)): self.battery._cache.clear() data = self.battery.GetPowerData() check = { - 'system_total': 2000.0, - 'per_package': { - 'test_package1': {'uid': '1000', 'data': [1.0]}, - 'test_package2': {'uid': '1001', 'data': [2.0]} - } + 'system_total': 2000.0, + 'per_package': { + 'test_package1': { + 'uid': '1000', + 'data': [1.0] + }, + 'test_package2': { + 'uid': '1001', + 'data': [2.0] + } + } } self.assertEqual(data, check) class BatteryUtilsChargeDevice(BatteryUtilsTest): - @mock.patch('time.sleep', mock.Mock()) def testChargeDeviceToLevel_pass(self): - with self.assertCalls( - (self.call.battery.SetCharging(True)), - (self.call.battery.GetBatteryInfo(), {'level': '50'}), - (self.call.battery.GetBatteryInfo(), {'level': '100'})): + with self.assertCalls((self.call.battery.SetCharging(True)), + (self.call.battery.GetBatteryInfo(), { + 'level': '50' + }), (self.call.battery.GetBatteryInfo(), { + 'level': '100' + })): self.battery.ChargeDeviceToLevel(95) @mock.patch('time.sleep', mock.Mock()) def testChargeDeviceToLevel_failureSame(self): - with self.assertCalls( - (self.call.battery.SetCharging(True)), - (self.call.battery.GetBatteryInfo(), {'level': '50'}), - (self.call.battery.GetBatteryInfo(), {'level': '50'}), - - (self.call.battery.GetBatteryInfo(), {'level': '50'})): + with self.assertCalls((self.call.battery.SetCharging(True)), + (self.call.battery.GetBatteryInfo(), { + 'level': '50' + }), (self.call.battery.GetBatteryInfo(), { + 'level': '50' + }), (self.call.battery.GetBatteryInfo(), { + 'level': '50' + })): with self.assertRaises(device_errors.DeviceChargingError): old_max = battery_utils._MAX_CHARGE_ERROR try: @@ -274,11 +303,14 @@ class BatteryUtilsChargeDevice(BatteryUtilsTest): @mock.patch('time.sleep', mock.Mock()) def testChargeDeviceToLevel_failureDischarge(self): - with self.assertCalls( - (self.call.battery.SetCharging(True)), - (self.call.battery.GetBatteryInfo(), {'level': '50'}), - (self.call.battery.GetBatteryInfo(), {'level': '49'}), - (self.call.battery.GetBatteryInfo(), {'level': '48'})): + with self.assertCalls((self.call.battery.SetCharging(True)), + (self.call.battery.GetBatteryInfo(), { + 'level': '50' + }), (self.call.battery.GetBatteryInfo(), { + 'level': '49' + }), (self.call.battery.GetBatteryInfo(), { + 'level': '48' + })): with self.assertRaises(device_errors.DeviceChargingError): old_max = battery_utils._MAX_CHARGE_ERROR try: @@ -289,158 +321,167 @@ class BatteryUtilsChargeDevice(BatteryUtilsTest): class BatteryUtilsDischargeDevice(BatteryUtilsTest): - @mock.patch('time.sleep', mock.Mock()) def testDischargeDevice_exact(self): - with self.assertCalls( - (self.call.battery.GetBatteryInfo(), {'level': '100'}), - (self.call.battery._HardwareSetCharging(False)), - (self.call.battery._HardwareSetCharging(True)), - (self.call.battery.GetBatteryInfo(), {'level': '99'})): + with self.assertCalls((self.call.battery.GetBatteryInfo(), { + 'level': '100' + }), (self.call.battery._HardwareSetCharging(False)), + (self.call.battery._HardwareSetCharging(True)), + (self.call.battery.GetBatteryInfo(), { + 'level': '99' + })): self.battery._DischargeDevice(1) @mock.patch('time.sleep', mock.Mock()) def testDischargeDevice_over(self): - with self.assertCalls( - (self.call.battery.GetBatteryInfo(), {'level': '100'}), - (self.call.battery._HardwareSetCharging(False)), - (self.call.battery._HardwareSetCharging(True)), - (self.call.battery.GetBatteryInfo(), {'level': '50'})): + with self.assertCalls((self.call.battery.GetBatteryInfo(), { + 'level': '100' + }), (self.call.battery._HardwareSetCharging(False)), + (self.call.battery._HardwareSetCharging(True)), + (self.call.battery.GetBatteryInfo(), { + 'level': '50' + })): self.battery._DischargeDevice(1) @mock.patch('time.sleep', mock.Mock()) def testDischargeDevice_takeslong(self): - with self.assertCalls( - (self.call.battery.GetBatteryInfo(), {'level': '100'}), - (self.call.battery._HardwareSetCharging(False)), - (self.call.battery._HardwareSetCharging(True)), - (self.call.battery.GetBatteryInfo(), {'level': '100'}), - (self.call.battery._HardwareSetCharging(False)), - (self.call.battery._HardwareSetCharging(True)), - (self.call.battery.GetBatteryInfo(), {'level': '99'}), - (self.call.battery._HardwareSetCharging(False)), - (self.call.battery._HardwareSetCharging(True)), - (self.call.battery.GetBatteryInfo(), {'level': '98'}), - (self.call.battery._HardwareSetCharging(False)), - (self.call.battery._HardwareSetCharging(True)), - (self.call.battery.GetBatteryInfo(), {'level': '97'})): + with self.assertCalls((self.call.battery.GetBatteryInfo(), { + 'level': '100' + }), (self.call.battery._HardwareSetCharging(False)), + (self.call.battery._HardwareSetCharging(True)), + (self.call.battery.GetBatteryInfo(), { + 'level': '100' + }), (self.call.battery._HardwareSetCharging(False)), + (self.call.battery._HardwareSetCharging(True)), + (self.call.battery.GetBatteryInfo(), { + 'level': '99' + }), (self.call.battery._HardwareSetCharging(False)), + (self.call.battery._HardwareSetCharging(True)), + (self.call.battery.GetBatteryInfo(), { + 'level': '98' + }), (self.call.battery._HardwareSetCharging(False)), + (self.call.battery._HardwareSetCharging(True)), + (self.call.battery.GetBatteryInfo(), { + 'level': '97' + })): self.battery._DischargeDevice(3) @mock.patch('time.sleep', mock.Mock()) def testDischargeDevice_dischargeTooClose(self): - with self.assertCalls( - (self.call.battery.GetBatteryInfo(), {'level': '100'})): + with self.assertCalls((self.call.battery.GetBatteryInfo(), { + 'level': '100' + })): self.battery._DischargeDevice(99) @mock.patch('time.sleep', mock.Mock()) def testDischargeDevice_percentageOutOfBounds(self): - with self.assertCalls( - (self.call.battery.GetBatteryInfo(), {'level': '100'})): + with self.assertCalls((self.call.battery.GetBatteryInfo(), { + 'level': '100' + })): with self.assertRaises(ValueError): self.battery._DischargeDevice(100) - with self.assertCalls( - (self.call.battery.GetBatteryInfo(), {'level': '100'})): + with self.assertCalls((self.call.battery.GetBatteryInfo(), { + 'level': '100' + })): with self.assertRaises(ValueError): self.battery._DischargeDevice(0) class BatteryUtilsGetBatteryInfoTest(BatteryUtilsTest): - def testGetBatteryInfo_normal(self): with self.assertCalls( - (self.call.device.RunShellCommand( - ['dumpsys', 'battery'], check_return=True), - [ - 'Current Battery Service state:', - ' AC powered: false', - ' USB powered: true', - ' level: 100', - ' temperature: 321', - ])): - self.assertEquals( - { - 'AC powered': 'false', - 'USB powered': 'true', - 'level': '100', - 'temperature': '321', - }, - self.battery.GetBatteryInfo()) + (self.call.device.RunShellCommand(['dumpsys', 'battery'], + check_return=True), [ + 'Current Battery Service state:', + ' AC powered: false', + ' USB powered: true', + ' level: 100', + ' temperature: 321', + ])): + self.assertEquals({ + 'AC powered': 'false', + 'USB powered': 'true', + 'level': '100', + 'temperature': '321', + }, self.battery.GetBatteryInfo()) def testGetBatteryInfo_nothing(self): with self.assertCalls( - (self.call.device.RunShellCommand( - ['dumpsys', 'battery'], check_return=True), [])): + (self.call.device.RunShellCommand(['dumpsys', 'battery'], + check_return=True), [])): self.assertEquals({}, self.battery.GetBatteryInfo()) class BatteryUtilsGetChargingTest(BatteryUtilsTest): - def testGetCharging_usb(self): - with self.assertCall( - self.call.battery.GetBatteryInfo(), {'USB powered': 'true'}): + with self.assertCall(self.call.battery.GetBatteryInfo(), + {'USB powered': 'true'}): self.assertTrue(self.battery.GetCharging()) def testGetCharging_usbFalse(self): - with self.assertCall( - self.call.battery.GetBatteryInfo(), {'USB powered': 'false'}): + with self.assertCall(self.call.battery.GetBatteryInfo(), + {'USB powered': 'false'}): self.assertFalse(self.battery.GetCharging()) def testGetCharging_ac(self): - with self.assertCall( - self.call.battery.GetBatteryInfo(), {'AC powered': 'true'}): + with self.assertCall(self.call.battery.GetBatteryInfo(), + {'AC powered': 'true'}): self.assertTrue(self.battery.GetCharging()) def testGetCharging_wireless(self): - with self.assertCall( - self.call.battery.GetBatteryInfo(), {'Wireless powered': 'true'}): + with self.assertCall(self.call.battery.GetBatteryInfo(), + {'Wireless powered': 'true'}): self.assertTrue(self.battery.GetCharging()) def testGetCharging_unknown(self): - with self.assertCall( - self.call.battery.GetBatteryInfo(), {'level': '42'}): + with self.assertCall(self.call.battery.GetBatteryInfo(), {'level': '42'}): self.assertFalse(self.battery.GetCharging()) class BatteryUtilsLetBatteryCoolToTemperatureTest(BatteryUtilsTest): - @mock.patch('time.sleep', mock.Mock()) def testLetBatteryCoolToTemperature_startUnder(self): self.battery._cache['profile'] = self._NEXUS_6 - with self.assertCalls( - (self.call.battery.EnableBatteryUpdates(), []), - (self.call.battery.GetBatteryInfo(), {'temperature': '500'})): + with self.assertCalls((self.call.battery.EnableBatteryUpdates(), []), + (self.call.battery.GetBatteryInfo(), { + 'temperature': '500' + })): self.battery.LetBatteryCoolToTemperature(600) @mock.patch('time.sleep', mock.Mock()) def testLetBatteryCoolToTemperature_startOver(self): self.battery._cache['profile'] = self._NEXUS_6 - with self.assertCalls( - (self.call.battery.EnableBatteryUpdates(), []), - (self.call.battery.GetBatteryInfo(), {'temperature': '500'}), - (self.call.battery.GetBatteryInfo(), {'temperature': '400'})): + with self.assertCalls((self.call.battery.EnableBatteryUpdates(), []), + (self.call.battery.GetBatteryInfo(), { + 'temperature': '500' + }), (self.call.battery.GetBatteryInfo(), { + 'temperature': '400' + })): self.battery.LetBatteryCoolToTemperature(400) @mock.patch('time.sleep', mock.Mock()) def testLetBatteryCoolToTemperature_nexus5Hot(self): self.battery._cache['profile'] = self._NEXUS_5 - with self.assertCalls( - (self.call.battery.EnableBatteryUpdates(), []), - (self.call.battery.GetBatteryInfo(), {'temperature': '500'}), - (self.call.battery._DischargeDevice(1), []), - (self.call.battery.GetBatteryInfo(), {'temperature': '400'})): + with self.assertCalls((self.call.battery.EnableBatteryUpdates(), []), + (self.call.battery.GetBatteryInfo(), { + 'temperature': '500' + }), (self.call.battery._DischargeDevice(1), []), + (self.call.battery.GetBatteryInfo(), { + 'temperature': '400' + })): self.battery.LetBatteryCoolToTemperature(400) @mock.patch('time.sleep', mock.Mock()) def testLetBatteryCoolToTemperature_nexus5Cool(self): self.battery._cache['profile'] = self._NEXUS_5 - with self.assertCalls( - (self.call.battery.EnableBatteryUpdates(), []), - (self.call.battery.GetBatteryInfo(), {'temperature': '400'})): + with self.assertCalls((self.call.battery.EnableBatteryUpdates(), []), + (self.call.battery.GetBatteryInfo(), { + 'temperature': '400' + })): self.battery.LetBatteryCoolToTemperature(400) class BatteryUtilsSupportsFuelGaugeTest(BatteryUtilsTest): - def testSupportsFuelGauge_false(self): self.battery._cache['profile'] = self._NEXUS_5 self.assertFalse(self.battery.SupportsFuelGauge()) @@ -459,7 +500,6 @@ class BatteryUtilsSupportsFuelGaugeTest(BatteryUtilsTest): class BatteryUtilsGetFuelGaugeChargeCounterTest(BatteryUtilsTest): - def testGetFuelGaugeChargeCounter_noFuelGauge(self): self.battery._cache['profile'] = self._NEXUS_5 with self.assertRaises(device_errors.CommandFailedError): @@ -467,21 +507,19 @@ class BatteryUtilsGetFuelGaugeChargeCounterTest(BatteryUtilsTest): def testGetFuelGaugeChargeCounter_fuelGaugePresent(self): self.battery._cache['profile'] = self._NEXUS_6 - with self.assertCalls( - (self.call.battery.SupportsFuelGauge(), True), - (self.call.device.ReadFile(mock.ANY), '123')): + with self.assertCalls((self.call.battery.SupportsFuelGauge(), True), + (self.call.device.ReadFile(mock.ANY), '123')): self.assertEqual(self.battery.GetFuelGaugeChargeCounter(), 123) class BatteryUtilsSetCharging(BatteryUtilsTest): - @mock.patch('time.sleep', mock.Mock()) def testSetCharging_softwareSetTrue(self): self.battery._cache['profile'] = self._NEXUS_6 with self.assertCalls( (self.call.battery.GetCharging(), False), - (self.call.device.RunShellCommand( - ['dumpsys', 'battery', 'reset'], check_return=True), []), + (self.call.device.RunShellCommand(['dumpsys', 'battery', 'reset'], + check_return=True), []), (self.call.battery.GetCharging(), False), (self.call.device.RunShellCommand( ['dumpsys', 'battery'], check_return=True), ['UPDATES STOPPED']), @@ -504,18 +542,16 @@ class BatteryUtilsSetCharging(BatteryUtilsTest): @mock.patch('time.sleep', mock.Mock()) def testSetCharging_hardwareSetTrue(self): self.battery._cache['profile'] = self._NEXUS_5 - with self.assertCalls( - (self.call.battery.GetCharging(), False), - (self.call.battery._HardwareSetCharging(True))): + with self.assertCalls((self.call.battery.GetCharging(), False), + (self.call.battery._HardwareSetCharging(True))): self.battery.SetCharging(True) @mock.patch('time.sleep', mock.Mock()) def testSetCharging_hardwareSetFalse(self): self.battery._cache['profile'] = self._NEXUS_5 - with self.assertCalls( - (self.call.battery.GetCharging(), True), - (self.call.battery._ClearPowerData(), True), - (self.call.battery._HardwareSetCharging(False))): + with self.assertCalls((self.call.battery.GetCharging(), True), + (self.call.battery._ClearPowerData(), True), + (self.call.battery._HardwareSetCharging(False))): self.battery.SetCharging(False) def testSetCharging_expectedStateAlreadyTrue(self): @@ -528,15 +564,13 @@ class BatteryUtilsSetCharging(BatteryUtilsTest): class BatteryUtilsPowerMeasurement(BatteryUtilsTest): - def testPowerMeasurement_hardware(self): self.battery._cache['profile'] = self._NEXUS_5 - with self.assertCalls( - (self.call.battery.GetCharging(), True), - (self.call.battery._ClearPowerData(), True), - (self.call.battery._HardwareSetCharging(False)), - (self.call.battery.GetCharging(), False), - (self.call.battery._HardwareSetCharging(True))): + with self.assertCalls((self.call.battery.GetCharging(), True), + (self.call.battery._ClearPowerData(), True), + (self.call.battery._HardwareSetCharging(False)), + (self.call.battery.GetCharging(), False), + (self.call.battery._HardwareSetCharging(True))): with self.battery.PowerMeasurement(): pass @@ -552,8 +586,8 @@ class BatteryUtilsPowerMeasurement(BatteryUtilsTest): ['dumpsys', 'battery', 'set', 'usb', '0'], check_return=True), []), (self.call.battery.GetCharging(), False), (self.call.battery.GetCharging(), False), - (self.call.device.RunShellCommand( - ['dumpsys', 'battery', 'reset'], check_return=True), []), + (self.call.device.RunShellCommand(['dumpsys', 'battery', 'reset'], + check_return=True), []), (self.call.battery.GetCharging(), False), (self.call.device.RunShellCommand( ['dumpsys', 'battery'], check_return=True), ['UPDATES STOPPED']), @@ -563,30 +597,25 @@ class BatteryUtilsPowerMeasurement(BatteryUtilsTest): class BatteryUtilsDiscoverDeviceProfile(BatteryUtilsTest): - def testDiscoverDeviceProfile_known(self): - with self.patch_call(self.call.device.product_model, - return_value='Nexus 4'): + with self.patch_call( + self.call.device.product_model, return_value='Nexus 4'): self.battery._DiscoverDeviceProfile() self.assertListEqual(self.battery._cache['profile']['name'], ["Nexus 4"]) def testDiscoverDeviceProfile_unknown(self): - with self.patch_call(self.call.device.product_model, - return_value='Other'): + with self.patch_call(self.call.device.product_model, return_value='Other'): self.battery._DiscoverDeviceProfile() self.assertListEqual(self.battery._cache['profile']['name'], []) class BatteryUtilsClearPowerData(BatteryUtilsTest): - def testClearPowerData_preL(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=20): + with self.patch_call(self.call.device.build_version_sdk, return_value=20): self.assertFalse(self.battery._ClearPowerData()) def testClearPowerData_clearedL(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=22): + with self.patch_call(self.call.device.build_version_sdk, return_value=22): with self.assertCalls( (self.call.device.RunShellCommand( ['dumpsys', 'battery', 'set', 'usb', '1'], check_return=True), @@ -597,15 +626,14 @@ class BatteryUtilsClearPowerData(BatteryUtilsTest): ['dumpsys', 'batterystats', '--reset'], check_return=True), []), (self.call.device.RunShellCommand( ['dumpsys', 'batterystats', '--charged', '-c'], - check_return=True, large_output=True), []), - (self.call.device.RunShellCommand( - ['dumpsys', 'battery', 'reset'], check_return=True), [])): + check_return=True, + large_output=True), []), (self.call.device.RunShellCommand( + ['dumpsys', 'battery', 'reset'], check_return=True), [])): self.assertTrue(self.battery._ClearPowerData()) @mock.patch('time.sleep', mock.Mock()) def testClearPowerData_notClearedL(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=22): + with self.patch_call(self.call.device.build_version_sdk, return_value=22): with self.assertCalls( (self.call.device.RunShellCommand( ['dumpsys', 'battery', 'set', 'usb', '1'], check_return=True), @@ -616,28 +644,28 @@ class BatteryUtilsClearPowerData(BatteryUtilsTest): ['dumpsys', 'batterystats', '--reset'], check_return=True), []), (self.call.device.RunShellCommand( ['dumpsys', 'batterystats', '--charged', '-c'], - check_return=True, large_output=True), - ['9,1000,l,pwi,uid,0.0327']), + check_return=True, + large_output=True), ['9,1000,l,pwi,uid,0.0327']), (self.call.device.RunShellCommand( ['dumpsys', 'batterystats', '--reset'], check_return=True), []), (self.call.device.RunShellCommand( ['dumpsys', 'batterystats', '--charged', '-c'], - check_return=True, large_output=True), - ['9,1000,l,pwi,uid,0.0327']), + check_return=True, + large_output=True), ['9,1000,l,pwi,uid,0.0327']), (self.call.device.RunShellCommand( ['dumpsys', 'batterystats', '--reset'], check_return=True), []), (self.call.device.RunShellCommand( ['dumpsys', 'batterystats', '--charged', '-c'], - check_return=True, large_output=True), - ['9,1000,l,pwi,uid,0.0327']), + check_return=True, + large_output=True), ['9,1000,l,pwi,uid,0.0327']), (self.call.device.RunShellCommand( ['dumpsys', 'batterystats', '--reset'], check_return=True), []), (self.call.device.RunShellCommand( ['dumpsys', 'batterystats', '--charged', '-c'], - check_return=True, large_output=True), - ['9,1000,l,pwi,uid,0.0']), - (self.call.device.RunShellCommand( - ['dumpsys', 'battery', 'reset'], check_return=True), [])): + check_return=True, + large_output=True), ['9,1000,l,pwi,uid,0.0']), + (self.call.device.RunShellCommand(['dumpsys', 'battery', 'reset'], + check_return=True), [])): self.battery._ClearPowerData() diff --git a/catapult/devil/devil/android/constants/chrome.py b/catapult/devil/devil/android/constants/chrome.py index 36bd972e..a02ce1d3 100644 --- a/catapult/devil/devil/android/constants/chrome.py +++ b/catapult/devil/devil/android/constants/chrome.py @@ -5,48 +5,36 @@ import collections PackageInfo = collections.namedtuple( - 'PackageInfo', - ['package', 'activity', 'cmdline_file', 'devtools_socket']) + 'PackageInfo', ['package', 'activity', 'cmdline_file', 'devtools_socket']) PACKAGE_INFO = { - 'chrome_document': PackageInfo( - 'com.google.android.apps.chrome.document', - 'com.google.android.apps.chrome.document.ChromeLauncherActivity', - 'chrome-command-line', - 'chrome_devtools_remote'), - 'chrome': PackageInfo( - 'com.google.android.apps.chrome', - 'com.google.android.apps.chrome.Main', - 'chrome-command-line', - 'chrome_devtools_remote'), - 'chrome_beta': PackageInfo( - 'com.chrome.beta', - 'com.google.android.apps.chrome.Main', - 'chrome-command-line', - 'chrome_devtools_remote'), - 'chrome_stable': PackageInfo( - 'com.android.chrome', - 'com.google.android.apps.chrome.Main', - 'chrome-command-line', - 'chrome_devtools_remote'), - 'chrome_dev': PackageInfo( - 'com.chrome.dev', - 'com.google.android.apps.chrome.Main', - 'chrome-command-line', - 'chrome_devtools_remote'), - 'chrome_canary': PackageInfo( - 'com.chrome.canary', - 'com.google.android.apps.chrome.Main', - 'chrome-command-line', - 'chrome_devtools_remote'), - 'chromium': PackageInfo( - 'org.chromium.chrome', - 'com.google.android.apps.chrome.Main', - 'chrome-command-line', - 'chrome_devtools_remote'), - 'content_shell': PackageInfo( - 'org.chromium.content_shell_apk', - '.ContentShellActivity', - 'content-shell-command-line', - 'content_shell_devtools_remote'), + 'chrome_document': + PackageInfo( + 'com.google.android.apps.chrome.document', + 'com.google.android.apps.chrome.document.ChromeLauncherActivity', + 'chrome-command-line', 'chrome_devtools_remote'), + 'chrome': + PackageInfo('com.google.android.apps.chrome', + 'com.google.android.apps.chrome.Main', + 'chrome-command-line', 'chrome_devtools_remote'), + 'chrome_beta': + PackageInfo('com.chrome.beta', 'com.google.android.apps.chrome.Main', + 'chrome-command-line', 'chrome_devtools_remote'), + 'chrome_stable': + PackageInfo('com.android.chrome', 'com.google.android.apps.chrome.Main', + 'chrome-command-line', 'chrome_devtools_remote'), + 'chrome_dev': + PackageInfo('com.chrome.dev', 'com.google.android.apps.chrome.Main', + 'chrome-command-line', 'chrome_devtools_remote'), + 'chrome_canary': + PackageInfo('com.chrome.canary', 'com.google.android.apps.chrome.Main', + 'chrome-command-line', 'chrome_devtools_remote'), + 'chromium': + PackageInfo('org.chromium.chrome', + 'com.google.android.apps.chrome.Main', + 'chrome-command-line', 'chrome_devtools_remote'), + 'content_shell': + PackageInfo('org.chromium.content_shell_apk', '.ContentShellActivity', + 'content-shell-command-line', + 'content_shell_devtools_remote'), } diff --git a/catapult/devil/devil/android/constants/webapk.py b/catapult/devil/devil/android/constants/webapk.py index 5a17e724..64241681 100644 --- a/catapult/devil/devil/android/constants/webapk.py +++ b/catapult/devil/devil/android/constants/webapk.py @@ -3,4 +3,3 @@ # found in the LICENSE file. WEBAPK_MAIN_ACTIVITY = 'org.chromium.webapk.shell_apk.MainActivity' - diff --git a/catapult/devil/devil/android/cpu_temperature.py b/catapult/devil/devil/android/cpu_temperature.py index 58ce87a0..7fa724c6 100644 --- a/catapult/devil/devil/android/cpu_temperature.py +++ b/catapult/devil/devil/android/cpu_temperature.py @@ -64,7 +64,6 @@ _DEVICE_THERMAL_INFORMATION = { class CpuTemperature(object): - def __init__(self, device): """CpuTemperature constructor. @@ -83,7 +82,7 @@ class CpuTemperature(object): """Init the current devices thermal information. """ self._device_info = _DEVICE_THERMAL_INFORMATION.get( - self._device.build_product) + self._device.build_product) def IsSupported(self): """Check if the current device is supported. diff --git a/catapult/devil/devil/android/cpu_temperature_test.py b/catapult/devil/devil/android/cpu_temperature_test.py index f0f99de0..8d082bb9 100644 --- a/catapult/devil/devil/android/cpu_temperature_test.py +++ b/catapult/devil/devil/android/cpu_temperature_test.py @@ -22,7 +22,6 @@ with devil_env.SysPath(devil_env.PYMOCK_PATH): class CpuTemperatureTest(mock_calls.TestCase): - @mock.patch('devil.android.perf.perf_control.PerfControl', mock.Mock()) def setUp(self): # Mock the device @@ -36,7 +35,6 @@ class CpuTemperatureTest(mock_calls.TestCase): class CpuTemperatureInitTest(unittest.TestCase): - @mock.patch('devil.android.perf.perf_control.PerfControl', mock.Mock()) def testInitWithDeviceUtil(self): d = mock.Mock(spec=device_utils.DeviceUtils) @@ -52,7 +50,6 @@ class CpuTemperatureInitTest(unittest.TestCase): class CpuTemperatureGetThermalDeviceInformationTest(CpuTemperatureTest): - @mock.patch('devil.android.perf.perf_control.PerfControl', mock.Mock()) def testGetThermalDeviceInformation_noneWhenIncorrectLabel(self): invalid_device = mock.Mock(spec=device_utils.DeviceUtils) @@ -78,7 +75,6 @@ class CpuTemperatureGetThermalDeviceInformationTest(CpuTemperatureTest): class CpuTemperatureIsSupportedTest(CpuTemperatureTest): - @mock.patch('devil.android.perf.perf_control.PerfControl', mock.Mock()) def testIsSupported_returnsTrue(self): d = mock.Mock(spec=device_utils.DeviceUtils) @@ -98,8 +94,10 @@ class CpuTemperatureIsSupportedTest(CpuTemperatureTest): class CpuTemperatureLetCpuCoolToTemperatureTest(CpuTemperatureTest): # Return values for the mock side effect - cooling_down0 = ([45000 for _ in range(8)] + [43000 for _ in range(8)] + - [41000 for _ in range(8)]) + cooling_down0 = ( + [45000 + for _ in range(8)] + [43000 + for _ in range(8)] + [41000 for _ in range(8)]) @mock.patch('time.sleep', mock.Mock()) def testLetBatteryCoolToTemperature_coolWithin24Calls(self): diff --git a/catapult/devil/devil/android/crash_handler.py b/catapult/devil/devil/android/crash_handler.py index 028e787d..9db5444b 100644 --- a/catapult/devil/devil/android/crash_handler.py +++ b/catapult/devil/devil/android/crash_handler.py @@ -2,7 +2,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - import logging from devil import base_error @@ -33,7 +32,7 @@ def RetryOnSystemCrash(f, device, retries=3): except device_errors.DeviceUnreachableError: if num_try > retries: logger.error('%d consecutive device crashes. No longer retrying.', - num_try) + num_try) raise try: logger.warning('Device is unreachable. Waiting for recovery...') diff --git a/catapult/devil/devil/android/crash_handler_devicetest.py b/catapult/devil/devil/android/crash_handler_devicetest.py index 6365104d..68c0d3ad 100755 --- a/catapult/devil/devil/android/crash_handler_devicetest.py +++ b/catapult/devil/devil/android/crash_handler_devicetest.py @@ -9,7 +9,11 @@ import unittest if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', ))) + os.path.abspath(os.path.join( + os.path.dirname(__file__), + '..', + '..', + ))) from devil.android import crash_handler from devil.android import device_errors @@ -22,7 +26,6 @@ from devil.utils import timeout_retry class DeviceCrashTest(device_test_case.DeviceTestCase): - def setUp(self): super(DeviceCrashTest, self).setUp() self.device = device_utils.DeviceUtils(self.serial) @@ -34,13 +37,16 @@ class DeviceCrashTest(device_test_case.DeviceTestCase): trigger_text = 'hello world' def victim(): - trigger_cmd = 'echo -n %s > %s; sleep 20' % ( - cmd_helper.SingleQuote(trigger_text), - cmd_helper.SingleQuote(trigger_file.name)) + trigger_cmd = 'echo -n %s > %s; sleep 20' % (cmd_helper.SingleQuote( + trigger_text), cmd_helper.SingleQuote(trigger_file.name)) crash_handler.RetryOnSystemCrash( lambda d: d.RunShellCommand( - trigger_cmd, shell=True, check_return=True, retries=1, - as_root=True, timeout=180), + trigger_cmd, + shell=True, + check_return=True, + retries=1, + as_root=True, + timeout=180), device=self.device) self.assertEquals( trigger_text, @@ -60,11 +66,12 @@ class DeviceCrashTest(device_test_case.DeviceTestCase): return False self.device.adb.Shell( 'echo c > /proc/sysrq-trigger', - expect_status=None, timeout=60, retries=0) + expect_status=None, + timeout=60, + retries=0) return True - self.assertEquals([True, True], - reraiser_thread.RunAsync([crasher, victim])) + self.assertEquals([True, True], reraiser_thread.RunAsync([crasher, victim])) if __name__ == '__main__': diff --git a/catapult/devil/devil/android/decorators.py b/catapult/devil/devil/android/decorators.py index 93e10544..0b3778aa 100644 --- a/catapult/devil/devil/android/decorators.py +++ b/catapult/devil/devil/android/decorators.py @@ -1,7 +1,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Function/method decorators that provide timeout and retry logic. """ @@ -19,9 +18,11 @@ DEFAULT_TIMEOUT_ATTR = '_default_timeout' DEFAULT_RETRIES_ATTR = '_default_retries' -def _TimeoutRetryWrapper( - f, timeout_func, retries_func, retry_if_func=timeout_retry.AlwaysRetry, - pass_values=False): +def _TimeoutRetryWrapper(f, + timeout_func, + retries_func, + retry_if_func=timeout_retry.AlwaysRetry, + pass_values=False): """ Wraps a funcion with timeout and retry handling logic. Args: @@ -34,6 +35,7 @@ def _TimeoutRetryWrapper( Returns: The wrapped function. """ + @functools.wraps(f) def timeout_retry_wrapper(*args, **kwargs): timeout = timeout_func(*args, **kwargs) @@ -45,22 +47,24 @@ def _TimeoutRetryWrapper( @functools.wraps(f) def impl(): return f(*args, **kwargs) + try: if timeout_retry.CurrentTimeoutThreadGroup(): # Don't wrap if there's already an outer timeout thread. return impl() else: - desc = '%s(%s)' % (f.__name__, ', '.join(itertools.chain( - (str(a) for a in args), - ('%s=%s' % (k, str(v)) for k, v in kwargs.iteritems())))) - return timeout_retry.Run(impl, timeout, retries, desc=desc, - retry_if_func=retry_if_func) + desc = '%s(%s)' % (f.__name__, ', '.join( + itertools.chain( + (str(a) for a in args), + ('%s=%s' % (k, str(v)) for k, v in kwargs.iteritems())))) + return timeout_retry.Run( + impl, timeout, retries, desc=desc, retry_if_func=retry_if_func) except reraiser_thread.TimeoutError as e: - raise device_errors.CommandTimeoutError(str(e)), None, ( - sys.exc_info()[2]) + raise device_errors.CommandTimeoutError(str(e)), None, (sys.exc_info()[2]) except cmd_helper.TimeoutError as e: - raise device_errors.CommandTimeoutError(str(e), output=e.output), None, ( - sys.exc_info()[2]) + raise device_errors.CommandTimeoutError( + str(e), output=e.output), None, (sys.exc_info()[2]) + return timeout_retry_wrapper @@ -90,11 +94,13 @@ def WithTimeoutAndConditionalRetries(retry_if_func): Returns: The actual decorator. """ + def decorator(f): get_timeout = lambda *a, **kw: kw['timeout'] get_retries = lambda *a, **kw: kw['retries'] return _TimeoutRetryWrapper( f, get_timeout, get_retries, retry_if_func=retry_if_func) + return decorator @@ -111,10 +117,12 @@ def WithExplicitTimeoutAndRetries(timeout, retries): Returns: The actual decorator. """ + def decorator(f): get_timeout = lambda *a, **kw: timeout get_retries = lambda *a, **kw: retries return _TimeoutRetryWrapper(f, get_timeout, get_retries) + return decorator @@ -134,17 +142,18 @@ def WithTimeoutAndRetriesDefaults(default_timeout, default_retries): Returns: The actual decorator. """ + def decorator(f): get_timeout = lambda *a, **kw: kw.get('timeout', default_timeout) get_retries = lambda *a, **kw: kw.get('retries', default_retries) return _TimeoutRetryWrapper(f, get_timeout, get_retries, pass_values=True) + return decorator -def WithTimeoutAndRetriesFromInstance( - default_timeout_name=DEFAULT_TIMEOUT_ATTR, - default_retries_name=DEFAULT_RETRIES_ATTR, - min_default_timeout=None): +def WithTimeoutAndRetriesFromInstance(default_timeout_name=DEFAULT_TIMEOUT_ATTR, + default_retries_name=DEFAULT_RETRIES_ATTR, + min_default_timeout=None): """Returns a decorator that handles timeouts and retries. The provided |default_timeout_name| and |default_retries_name| are used to @@ -162,6 +171,7 @@ def WithTimeoutAndRetriesFromInstance( Returns: The actual decorator. """ + def decorator(f): def get_timeout(inst, *_args, **kwargs): ret = getattr(inst, default_timeout_name) @@ -171,6 +181,7 @@ def WithTimeoutAndRetriesFromInstance( def get_retries(inst, *_args, **kwargs): return kwargs.get('retries', getattr(inst, default_retries_name)) + return _TimeoutRetryWrapper(f, get_timeout, get_retries, pass_values=True) - return decorator + return decorator diff --git a/catapult/devil/devil/android/decorators_test.py b/catapult/devil/devil/android/decorators_test.py index f60953e1..994e35e8 100644 --- a/catapult/devil/devil/android/decorators_test.py +++ b/catapult/devil/devil/android/decorators_test.py @@ -1,7 +1,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Unit tests for decorators.py. """ @@ -54,6 +53,7 @@ class DecoratorsTest(unittest.TestCase): def testFunctionDecoratorRequiresParams(self): """Tests that the base decorator requires timeout and retries params.""" + @decorators.WithTimeoutAndRetries def requiresExplicitTimeoutAndRetries(timeout=None, retries=None): return (timeout, retries) @@ -66,14 +66,14 @@ class DecoratorsTest(unittest.TestCase): requiresExplicitTimeoutAndRetries(retries=0) expected_timeout = 10 expected_retries = 1 - (actual_timeout, actual_retries) = ( - requiresExplicitTimeoutAndRetries(timeout=expected_timeout, - retries=expected_retries)) + (actual_timeout, actual_retries) = (requiresExplicitTimeoutAndRetries( + timeout=expected_timeout, retries=expected_retries)) self.assertEquals(expected_timeout, actual_timeout) self.assertEquals(expected_retries, actual_retries) def testFunctionDecoratorTranslatesReraiserExceptions(self): """Tests that the explicit decorator translates reraiser exceptions.""" + @decorators.WithTimeoutAndRetries def alwaysRaisesProvidedException(exception, timeout=None, retries=None): raise exception @@ -81,8 +81,7 @@ class DecoratorsTest(unittest.TestCase): exception_desc = 'Reraiser thread timeout error' with self.assertRaises(device_errors.CommandTimeoutError) as e: alwaysRaisesProvidedException( - reraiser_thread.TimeoutError(exception_desc), - timeout=10, retries=1) + reraiser_thread.TimeoutError(exception_desc), timeout=10, retries=1) self.assertEquals(exception_desc, str(e.exception)) def testConditionalRetriesDecoratorRetries(self): @@ -158,6 +157,7 @@ class DecoratorsTest(unittest.TestCase): def testDefaultsFunctionDecoratorPassesValues(self): """Tests that the defaults decorator passes timeout and retries kwargs.""" + @decorators.WithTimeoutAndRetriesDefaults(30, 10) def alwaysReturnsTimeouts(timeout=None, retries=None): return timeout @@ -174,6 +174,7 @@ class DecoratorsTest(unittest.TestCase): def testDefaultsFunctionDecoratorTranslatesReraiserExceptions(self): """Tests that the explicit decorator translates reraiser exceptions.""" + @decorators.WithTimeoutAndRetriesDefaults(30, 10) def alwaysRaisesProvidedException(exception, timeout=None, retries=None): raise exception @@ -215,6 +216,7 @@ class DecoratorsTest(unittest.TestCase): def testExplicitDecoratorTranslatesReraiserExceptions(self): """Tests that the explicit decorator translates reraiser exceptions.""" + @decorators.WithExplicitTimeoutAndRetries(30, 10) def alwaysRaisesProvidedException(exception): raise exception @@ -228,7 +230,9 @@ class DecoratorsTest(unittest.TestCase): class _MethodDecoratorTestObject(object): """An object suitable for testing the method decorator.""" - def __init__(self, test_case, default_timeout=_DEFAULT_TIMEOUT, + def __init__(self, + test_case, + default_timeout=_DEFAULT_TIMEOUT, default_retries=_DEFAULT_RETRIES): self._test_case = test_case self.default_timeout = default_timeout @@ -239,23 +243,23 @@ class DecoratorsTest(unittest.TestCase): 'requiresExplicitTimeoutAndRetries': 0, } - @decorators.WithTimeoutAndRetriesFromInstance( - 'default_timeout', 'default_retries') + @decorators.WithTimeoutAndRetriesFromInstance('default_timeout', + 'default_retries') def alwaysTimesOut(self, timeout=None, retries=None): self.function_call_counters['alwaysTimesOut'] += 1 time.sleep(100) self._test_case.assertFalse(True, msg='Failed to time out?') - @decorators.WithTimeoutAndRetriesFromInstance( - 'default_timeout', 'default_retries') + @decorators.WithTimeoutAndRetriesFromInstance('default_timeout', + 'default_retries') def alwaysRaisesCommandFailedError(self, timeout=None, retries=None): self.function_call_counters['alwaysRaisesCommandFailedError'] += 1 raise device_errors.CommandFailedError('testCommand failed') # pylint: disable=no-self-use - @decorators.WithTimeoutAndRetriesFromInstance( - 'default_timeout', 'default_retries') + @decorators.WithTimeoutAndRetriesFromInstance('default_timeout', + 'default_retries') def alwaysReturnsTimeout(self, timeout=None, retries=None): return timeout @@ -264,14 +268,16 @@ class DecoratorsTest(unittest.TestCase): def alwaysReturnsTimeoutWithMin(self, timeout=None, retries=None): return timeout - @decorators.WithTimeoutAndRetriesFromInstance( - 'default_timeout', 'default_retries') + @decorators.WithTimeoutAndRetriesFromInstance('default_timeout', + 'default_retries') def alwaysReturnsRetries(self, timeout=None, retries=None): return retries - @decorators.WithTimeoutAndRetriesFromInstance( - 'default_timeout', 'default_retries') - def alwaysRaisesProvidedException(self, exception, timeout=None, + @decorators.WithTimeoutAndRetriesFromInstance('default_timeout', + 'default_retries') + def alwaysRaisesProvidedException(self, + exception, + timeout=None, retries=None): raise exception @@ -327,6 +333,6 @@ class DecoratorsTest(unittest.TestCase): reraiser_thread.TimeoutError(exception_desc)) self.assertEquals(exception_desc, str(e.exception)) + if __name__ == '__main__': unittest.main(verbosity=2) - diff --git a/catapult/devil/devil/android/device_blacklist.py b/catapult/devil/devil/android/device_blacklist.py index 010e9965..7112204e 100644 --- a/catapult/devil/devil/android/device_blacklist.py +++ b/catapult/devil/devil/android/device_blacklist.py @@ -2,79 +2,8 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -import json -import logging -import os -import threading -import time +from devil.android import device_denylist -logger = logging.getLogger(__name__) - - -class Blacklist(object): - - def __init__(self, path): - self._blacklist_lock = threading.RLock() - self._path = path - - def Read(self): - """Reads the blacklist from the blacklist file. - - Returns: - A dict containing bad devices. - """ - with self._blacklist_lock: - blacklist = dict() - if not os.path.exists(self._path): - return blacklist - - try: - with open(self._path, 'r') as f: - blacklist = json.load(f) - except (IOError, ValueError) as e: - logger.warning('Unable to read blacklist: %s', str(e)) - os.remove(self._path) - - if not isinstance(blacklist, dict): - logger.warning('Ignoring %s: %s (a dict was expected instead)', - self._path, blacklist) - blacklist = dict() - - return blacklist - - def Write(self, blacklist): - """Writes the provided blacklist to the blacklist file. - - Args: - blacklist: list of bad devices to write to the blacklist file. - """ - with self._blacklist_lock: - with open(self._path, 'w') as f: - json.dump(blacklist, f) - - def Extend(self, devices, reason='unknown'): - """Adds devices to blacklist file. - - Args: - devices: list of bad devices to be added to the blacklist file. - reason: string specifying the reason for blacklist (eg: 'unauthorized') - """ - timestamp = time.time() - event_info = { - 'timestamp': timestamp, - 'reason': reason, - } - device_dicts = {device: event_info for device in devices} - logger.info('Adding %s to blacklist %s for reason: %s', - ','.join(devices), self._path, reason) - with self._blacklist_lock: - blacklist = self.Read() - blacklist.update(device_dicts) - self.Write(blacklist) - - def Reset(self): - """Erases the blacklist file if it exists.""" - logger.info('Resetting blacklist %s', self._path) - with self._blacklist_lock: - if os.path.exists(self._path): - os.remove(self._path) +# TODO(crbug.com/1097306): Remove this (and this file) once existing uses +# have switched to using device_denylist directly. +Blacklist = device_denylist.Denylist diff --git a/catapult/devil/devil/android/device_blacklist_test.py b/catapult/devil/devil/android/device_blacklist_test.py deleted file mode 100644 index bc44da55..00000000 --- a/catapult/devil/devil/android/device_blacklist_test.py +++ /dev/null @@ -1,38 +0,0 @@ -#! /usr/bin/env python -# Copyright 2016 The Chromium Authors. All rights reserved. -# Use of this source code is governed by a BSD-style license that can be -# found in the LICENSE file. - -import os -import tempfile -import unittest - -from devil.android import device_blacklist - - -class DeviceBlacklistTest(unittest.TestCase): - - def testBlacklistFileDoesNotExist(self): - with tempfile.NamedTemporaryFile() as blacklist_file: - # Allow the temporary file to be deleted. - pass - - test_blacklist = device_blacklist.Blacklist(blacklist_file.name) - self.assertEquals({}, test_blacklist.Read()) - - def testBlacklistFileIsEmpty(self): - try: - with tempfile.NamedTemporaryFile(delete=False) as blacklist_file: - # Allow the temporary file to be closed. - pass - - test_blacklist = device_blacklist.Blacklist(blacklist_file.name) - self.assertEquals({}, test_blacklist.Read()) - - finally: - if os.path.exists(blacklist_file.name): - os.remove(blacklist_file.name) - - -if __name__ == '__main__': - unittest.main() diff --git a/catapult/devil/devil/android/device_denylist.py b/catapult/devil/devil/android/device_denylist.py new file mode 100644 index 00000000..88b5969c --- /dev/null +++ b/catapult/devil/devil/android/device_denylist.py @@ -0,0 +1,79 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import json +import logging +import os +import threading +import time + +logger = logging.getLogger(__name__) + + +class Denylist(object): + def __init__(self, path): + self._denylist_lock = threading.RLock() + self._path = path + + def Read(self): + """Reads the denylist from the denylist file. + + Returns: + A dict containing bad devices. + """ + with self._denylist_lock: + denylist = dict() + if not os.path.exists(self._path): + return denylist + + try: + with open(self._path, 'r') as f: + denylist = json.load(f) + except (IOError, ValueError) as e: + logger.warning('Unable to read denylist: %s', str(e)) + os.remove(self._path) + + if not isinstance(denylist, dict): + logger.warning('Ignoring %s: %s (a dict was expected instead)', + self._path, denylist) + denylist = dict() + + return denylist + + def Write(self, denylist): + """Writes the provided denylist to the denylist file. + + Args: + denylist: list of bad devices to write to the denylist file. + """ + with self._denylist_lock: + with open(self._path, 'w') as f: + json.dump(denylist, f) + + def Extend(self, devices, reason='unknown'): + """Adds devices to denylist file. + + Args: + devices: list of bad devices to be added to the denylist file. + reason: string specifying the reason for denylist (eg: 'unauthorized') + """ + timestamp = time.time() + event_info = { + 'timestamp': timestamp, + 'reason': reason, + } + device_dicts = {device: event_info for device in devices} + logger.info('Adding %s to denylist %s for reason: %s', ','.join(devices), + self._path, reason) + with self._denylist_lock: + denylist = self.Read() + denylist.update(device_dicts) + self.Write(denylist) + + def Reset(self): + """Erases the denylist file if it exists.""" + logger.info('Resetting denylist %s', self._path) + with self._denylist_lock: + if os.path.exists(self._path): + os.remove(self._path) diff --git a/catapult/devil/devil/android/device_denylist_test.py b/catapult/devil/devil/android/device_denylist_test.py new file mode 100644 index 00000000..b5d79e83 --- /dev/null +++ b/catapult/devil/devil/android/device_denylist_test.py @@ -0,0 +1,37 @@ +#! /usr/bin/env python +# Copyright 2016 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import os +import tempfile +import unittest + +from devil.android import device_denylist + + +class DeviceDenylistTest(unittest.TestCase): + def testDenylistFileDoesNotExist(self): + with tempfile.NamedTemporaryFile() as denylist_file: + # Allow the temporary file to be deleted. + pass + + test_denylist = device_denylist.Denylist(denylist_file.name) + self.assertEquals({}, test_denylist.Read()) + + def testDenylistFileIsEmpty(self): + try: + with tempfile.NamedTemporaryFile(delete=False) as denylist_file: + # Allow the temporary file to be closed. + pass + + test_denylist = device_denylist.Denylist(denylist_file.name) + self.assertEquals({}, test_denylist.Read()) + + finally: + if os.path.exists(denylist_file.name): + os.remove(denylist_file.name) + + +if __name__ == '__main__': + unittest.main() diff --git a/catapult/devil/devil/android/device_errors.py b/catapult/devil/devil/android/device_errors.py index e6893a4f..d1454683 100644 --- a/catapult/devil/devil/android/device_errors.py +++ b/catapult/devil/devil/android/device_errors.py @@ -1,7 +1,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Exception classes raised by AdbWrapper and DeviceUtils. @@ -11,6 +10,7 @@ The class hierarchy for device exceptions is: +-- CommandFailedError | +-- AdbCommandFailedError | | +-- AdbShellCommandFailedError + | +-- AdbVersionError | +-- FastbootCommandFailedError | +-- DeviceVersionError | +-- DeviceChargingError @@ -48,7 +48,11 @@ class CommandFailedError(base_error.BaseError): class _BaseCommandFailedError(CommandFailedError): """Base Exception for adb and fastboot command failures.""" - def __init__(self, args, output, status=None, device_serial=None, + def __init__(self, + args, + output, + status=None, + device_serial=None, message=None): self.args = args self.output = output @@ -68,8 +72,7 @@ class _BaseCommandFailedError(CommandFailedError): def __eq__(self, other): return (super(_BaseCommandFailedError, self).__eq__(other) - and self.args == other.args - and self.output == other.output + and self.args == other.args and self.output == other.output and self.status == other.status) def __ne__(self, other): @@ -82,28 +85,42 @@ class _BaseCommandFailedError(CommandFailedError): result[:len(super_result)] = super_result # Update the args used to reconstruct this exception. - result[1] = ( - self.args, self.output, self.status, self.device_serial, self.message) + result[1] = (self.args, self.output, self.status, self.device_serial, + self.message) return tuple(result) class AdbCommandFailedError(_BaseCommandFailedError): """Exception for adb command failures.""" - def __init__(self, args, output, status=None, device_serial=None, + def __init__(self, + args, + output, + status=None, + device_serial=None, message=None): super(AdbCommandFailedError, self).__init__( - args, output, status=status, message=message, + args, + output, + status=status, + message=message, device_serial=device_serial) class FastbootCommandFailedError(_BaseCommandFailedError): """Exception for fastboot command failures.""" - def __init__(self, args, output, status=None, device_serial=None, + def __init__(self, + args, + output, + status=None, + device_serial=None, message=None): super(FastbootCommandFailedError, self).__init__( - args, output, status=status, message=message, + args, + output, + status=status, + message=message, device_serial=device_serial) @@ -114,13 +131,29 @@ class DeviceVersionError(CommandFailedError): super(DeviceVersionError, self).__init__(message, device_serial) +class AdbVersionError(CommandFailedError): + """Exception for running a command on an incompatible version of adb.""" + + def __init__(self, args, desc=None, actual_version=None, min_version=None): + adb_cmd = ' '.join(cmd_helper.SingleQuote(arg) for arg in args) + desc = desc or 'not supported' + if min_version: + desc += ' prior to %s' % min_version + if actual_version: + desc += ' (actual: %s)' % actual_version + super(AdbVersionError, + self).__init__(message='adb %s: %s' % (adb_cmd, desc)) + + class AdbShellCommandFailedError(AdbCommandFailedError): """Exception for shell command failures run via adb.""" def __init__(self, command, output, status, device_serial=None): self.command = command - segments = ['shell command run via adb failed on the device:\n', - ' command: %s\n' % command] + segments = [ + 'shell command run via adb failed on the device:\n', + ' command: %s\n' % command + ] segments.append(' exit status: %s\n' % status) if output: segments.append(' output:\n') @@ -133,7 +166,7 @@ class AdbShellCommandFailedError(AdbCommandFailedError): segments.append(" output: ''\n") message = ''.join(segments) super(AdbShellCommandFailedError, self).__init__( - ['shell', command], output, status, device_serial, message) + ['shell', command], output, status, device_serial, message) def __reduce__(self): """Support pickling.""" @@ -148,6 +181,7 @@ class AdbShellCommandFailedError(AdbCommandFailedError): class CommandTimeoutError(base_error.BaseError): """Exception for command timeouts.""" + def __init__(self, message, is_infra_error=False, output=None): super(CommandTimeoutError, self).__init__(message, is_infra_error) self.output = output @@ -171,8 +205,8 @@ class MultipleDevicesError(base_error.BaseError): def __init__(self, devices): parallel_devices = parallelizer.Parallelizer(devices) - descriptions = parallel_devices.pMap( - lambda d: d.build_description).pGet(None) + descriptions = parallel_devices.pMap(lambda d: d.build_description).pGet( + None) msg = ('More than one device available. Use -d/--device to select a device ' 'by serial.\n\nAvailable devices:\n') for d, desc in zip(devices, descriptions): diff --git a/catapult/devil/devil/android/device_errors_test.py b/catapult/devil/devil/android/device_errors_test.py index 68a4f167..5fc9e254 100755 --- a/catapult/devil/devil/android/device_errors_test.py +++ b/catapult/devil/devil/android/device_errors_test.py @@ -11,7 +11,6 @@ from devil.android import device_errors class DeviceErrorsTest(unittest.TestCase): - def assertIsPicklable(self, original): pickled = pickle.dumps(original) reconstructed = pickle.loads(pickled) @@ -19,7 +18,9 @@ class DeviceErrorsTest(unittest.TestCase): def testPicklable_AdbCommandFailedError(self): original = device_errors.AdbCommandFailedError( - ['these', 'are', 'adb', 'args'], 'adb failure output', status=':(', + ['these', 'are', 'adb', 'args'], + 'adb failure output', + status=':(', device_serial='0123456789abcdef') self.assertIsPicklable(original) @@ -29,18 +30,15 @@ class DeviceErrorsTest(unittest.TestCase): self.assertIsPicklable(original) def testPicklable_CommandFailedError(self): - original = device_errors.CommandFailedError( - 'sample command failed') + original = device_errors.CommandFailedError('sample command failed') self.assertIsPicklable(original) def testPicklable_CommandTimeoutError(self): - original = device_errors.CommandTimeoutError( - 'My fake command timed out :(') + original = device_errors.CommandTimeoutError('My fake command timed out :(') self.assertIsPicklable(original) def testPicklable_DeviceChargingError(self): - original = device_errors.DeviceChargingError( - 'Fake device failed to charge') + original = device_errors.DeviceChargingError('Fake device failed to charge') self.assertIsPicklable(original) def testPicklable_DeviceUnreachableError(self): @@ -49,8 +47,10 @@ class DeviceErrorsTest(unittest.TestCase): def testPicklable_FastbootCommandFailedError(self): original = device_errors.FastbootCommandFailedError( - ['these', 'are', 'fastboot', 'args'], 'fastboot failure output', - status=':(', device_serial='0123456789abcdef') + ['these', 'are', 'fastboot', 'args'], + 'fastboot failure output', + status=':(', + device_serial='0123456789abcdef') self.assertIsPicklable(original) def testPicklable_MultipleDevicesError(self): @@ -67,6 +67,5 @@ class DeviceErrorsTest(unittest.TestCase): self.assertIsPicklable(original) - if __name__ == '__main__': sys.exit(unittest.main()) diff --git a/catapult/devil/devil/android/device_list.py b/catapult/devil/devil/android/device_list.py index 0fbb0f15..cd631dbd 100644 --- a/catapult/devil/devil/android/device_list.py +++ b/catapult/devil/devil/android/device_list.py @@ -1,7 +1,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A module to keep track of devices across builds.""" import json @@ -26,8 +25,8 @@ def GetPersistentDeviceList(file_name): try: with open(file_name) as f: devices = json.load(f) - if not isinstance(devices, list) or not all(isinstance(d, basestring) - for d in devices): + if not isinstance(devices, list) or not all( + isinstance(d, basestring) for d in devices): logger.warning('Unrecognized device file format: %s', devices) return [] return [d for d in devices if d != '(error)'] diff --git a/catapult/devil/devil/android/device_signal.py b/catapult/devil/devil/android/device_signal.py index 2cec46d7..a7149b37 100644 --- a/catapult/devil/devil/android/device_signal.py +++ b/catapult/devil/devil/android/device_signal.py @@ -1,13 +1,11 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Defines constants for signals that should be supported on devices. Note: Obtained by running `kill -l` on a user device. """ - SIGHUP = 1 # Hangup SIGINT = 2 # Interrupt SIGQUIT = 3 # Quit diff --git a/catapult/devil/devil/android/device_temp_file.py b/catapult/devil/devil/android/device_temp_file.py index 74cc5099..84e51cb2 100644 --- a/catapult/devil/devil/android/device_temp_file.py +++ b/catapult/devil/devil/android/device_temp_file.py @@ -1,7 +1,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A temp file that automatically gets pushed and deleted from a device.""" # pylint: disable=W0622 @@ -19,7 +18,7 @@ logger = logging.getLogger(__name__) def _GenerateName(prefix, suffix, dir): - random_hex = hex(random.randint(0, 2 ** 52))[2:] + random_hex = hex(random.randint(0, 2**52))[2:] return posixpath.join(dir, '%s-%s%s' % (prefix, random_hex, suffix)) @@ -55,15 +54,17 @@ class DeviceTempFile(object): def close(self): """Deletes the temporary file from the device.""" + # ignore exception if the file is already gone. def delete_temporary_file(): try: - self._adb.Shell('rm -f %s' % self.name_quoted, expect_status=None) + self._adb.Shell( + 'rm -f %s' % self.name_quoted, expect_status=None, retries=0) except base_error.BaseError as e: # We don't really care, and stack traces clog up the log. # Log a warning and move on. - logger.warning('Failed to delete temporary file %s: %s', - self.name, str(e)) + logger.warning('Failed to delete temporary file %s: %s', self.name, + str(e)) # It shouldn't matter when the temp file gets deleted, so do so # asynchronously. @@ -101,6 +102,7 @@ class NamedDeviceTemporaryDirectory(object): def close(self): """Deletes the temporary directory from the device.""" + def delete_temporary_dir(): try: self._adb.Shell('rm -rf %s' % self.name, expect_status=None) diff --git a/catapult/devil/devil/android/device_test_case.py b/catapult/devil/devil/android/device_test_case.py index 1148b544..327d67a6 100644 --- a/catapult/devil/devil/android/device_test_case.py +++ b/catapult/devil/devil/android/device_test_case.py @@ -21,8 +21,7 @@ def PrepareDevices(*_args): try: d.WaitUntilFullyBooted(timeout=5, retries=0) live_devices.append(str(d)) - except (device_errors.CommandFailedError, - device_errors.CommandTimeoutError, + except (device_errors.CommandFailedError, device_errors.CommandTimeoutError, device_errors.DeviceUnreachableError): pass with _devices_lock: @@ -33,7 +32,6 @@ def PrepareDevices(*_args): class DeviceTestCase(unittest.TestCase): - def __init__(self, *args, **kwargs): super(DeviceTestCase, self).__init__(*args, **kwargs) self.serial = None diff --git a/catapult/devil/devil/android/device_utils.py b/catapult/devil/devil/android/device_utils.py index 6182a527..0a041edb 100644 --- a/catapult/devil/devil/android/device_utils.py +++ b/catapult/devil/devil/android/device_utils.py @@ -1,11 +1,7 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - -"""Provides a variety of device interactions based on adb. - -Eventually, this will be based on adb_wrapper. -""" +"""Provides a variety of device interactions based on adb.""" # pylint: disable=unused-argument import calendar @@ -42,7 +38,6 @@ from devil.android import md5sum from devil.android.sdk import adb_wrapper from devil.android.sdk import intent from devil.android.sdk import keyevent -from devil.android.sdk import split_select from devil.android.sdk import version_codes from devil.utils import host_utils from devil.utils import parallelizer @@ -64,7 +59,7 @@ _DEFAULT_TIMEOUT = 30 _DEFAULT_RETRIES = 3 # A sentinel object for default values -# TODO(jbudorick,perezju): revisit how default values are handled by +# TODO(jbudorick): revisit how default values are handled by # the timeout_retry decorators. DEFAULT = object() @@ -73,22 +68,29 @@ DEFAULT = object() # as_root param. _FORCE_SU = object() -_RECURSIVE_DIRECTORY_LIST_SCRIPT = """ - function list_subdirs() { - for f in "$1"/* ; +# Lists all files for the specified directories. +# In order to minimize data transfer, prints directories as absolute paths +# followed by files within that directory without their path. +_FILE_LIST_SCRIPT = """ + function list_files() { + for f in "$1"/{.,}* do - if [ -d "$f" ] ; + if [ "$f" == "." ] || [ "$f" == ".." ] || [ "$f" == "${1}/.*" ] \ + || [ "$f" == "${1}/*" ] then - if [ "$f" == "." ] || [ "$f" == ".." ] ; - then - continue ; - fi ; - echo "$f" ; - list_subdirs "$f" ; - fi ; - done ; - } ; - list_subdirs %s + continue + fi + base=${f##*/} # Get the basename for the file, dropping the path. + echo "$base" + done + } + for dir in %s + do + if [ -d "$dir" ]; then + echo "$dir" + list_files "$dir" + fi + done """ _RESTART_ADBD_SCRIPT = """ @@ -103,74 +105,78 @@ _RESTART_ADBD_SCRIPT = """ """ # Not all permissions can be set. -_PERMISSIONS_BLACKLIST_RE = re.compile('|'.join(fnmatch.translate(p) for p in [ - 'android.permission.ACCESS_LOCATION_EXTRA_COMMANDS', - 'android.permission.ACCESS_MOCK_LOCATION', - 'android.permission.ACCESS_NETWORK_STATE', - 'android.permission.ACCESS_NOTIFICATION_POLICY', - 'android.permission.ACCESS_VR_STATE', - 'android.permission.ACCESS_WIFI_STATE', - 'android.permission.AUTHENTICATE_ACCOUNTS', - 'android.permission.BLUETOOTH', - 'android.permission.BLUETOOTH_ADMIN', - 'android.permission.BROADCAST_STICKY', - 'android.permission.CHANGE_NETWORK_STATE', - 'android.permission.CHANGE_WIFI_MULTICAST_STATE', - 'android.permission.CHANGE_WIFI_STATE', - 'android.permission.DISABLE_KEYGUARD', - 'android.permission.DOWNLOAD_WITHOUT_NOTIFICATION', - 'android.permission.EXPAND_STATUS_BAR', - 'android.permission.FOREGROUND_SERVICE', - 'android.permission.GET_PACKAGE_SIZE', - 'android.permission.INSTALL_SHORTCUT', - 'android.permission.INJECT_EVENTS', - 'android.permission.INTERNET', - 'android.permission.KILL_BACKGROUND_PROCESSES', - 'android.permission.MANAGE_ACCOUNTS', - 'android.permission.MODIFY_AUDIO_SETTINGS', - 'android.permission.NFC', - 'android.permission.READ_SYNC_SETTINGS', - 'android.permission.READ_SYNC_STATS', - 'android.permission.RECEIVE_BOOT_COMPLETED', - 'android.permission.RECORD_VIDEO', - 'android.permission.REORDER_TASKS', - 'android.permission.REQUEST_INSTALL_PACKAGES', - 'android.permission.RESTRICTED_VR_ACCESS', - 'android.permission.RUN_INSTRUMENTATION', - 'android.permission.SET_ALARM', - 'android.permission.SET_TIME_ZONE', - 'android.permission.SET_WALLPAPER', - 'android.permission.SET_WALLPAPER_HINTS', - 'android.permission.TRANSMIT_IR', - 'android.permission.USE_CREDENTIALS', - 'android.permission.USE_FINGERPRINT', - 'android.permission.VIBRATE', - 'android.permission.WAKE_LOCK', - 'android.permission.WRITE_SYNC_SETTINGS', - 'com.android.browser.permission.READ_HISTORY_BOOKMARKS', - 'com.android.browser.permission.WRITE_HISTORY_BOOKMARKS', - 'com.android.launcher.permission.INSTALL_SHORTCUT', - 'com.chrome.permission.DEVICE_EXTRAS', - 'com.google.android.apps.now.CURRENT_ACCOUNT_ACCESS', - 'com.google.android.c2dm.permission.RECEIVE', - 'com.google.android.providers.gsf.permission.READ_GSERVICES', - 'com.google.vr.vrcore.permission.VRCORE_INTERNAL', - 'com.sec.enterprise.knox.MDM_CONTENT_PROVIDER', - '*.permission.C2D_MESSAGE', - '*.permission.READ_WRITE_BOOKMARK_FOLDERS', - '*.TOS_ACKED', -])) +_PERMISSIONS_DENYLIST_RE = re.compile('|'.join( + fnmatch.translate(p) for p in [ + 'android.permission.ACCESS_LOCATION_EXTRA_COMMANDS', + 'android.permission.ACCESS_MOCK_LOCATION', + 'android.permission.ACCESS_NETWORK_STATE', + 'android.permission.ACCESS_NOTIFICATION_POLICY', + 'android.permission.ACCESS_VR_STATE', + 'android.permission.ACCESS_WIFI_STATE', + 'android.permission.AUTHENTICATE_ACCOUNTS', + 'android.permission.BLUETOOTH', + 'android.permission.BLUETOOTH_ADMIN', + 'android.permission.BROADCAST_STICKY', + 'android.permission.CHANGE_NETWORK_STATE', + 'android.permission.CHANGE_WIFI_MULTICAST_STATE', + 'android.permission.CHANGE_WIFI_STATE', + 'android.permission.DISABLE_KEYGUARD', + 'android.permission.DOWNLOAD_WITHOUT_NOTIFICATION', + 'android.permission.EXPAND_STATUS_BAR', + 'android.permission.FOREGROUND_SERVICE', + 'android.permission.GET_PACKAGE_SIZE', + 'android.permission.INSTALL_SHORTCUT', + 'android.permission.INJECT_EVENTS', + 'android.permission.INTERNET', + 'android.permission.KILL_BACKGROUND_PROCESSES', + 'android.permission.MANAGE_ACCOUNTS', + 'android.permission.MODIFY_AUDIO_SETTINGS', + 'android.permission.NFC', + 'android.permission.READ_SYNC_SETTINGS', + 'android.permission.READ_SYNC_STATS', + 'android.permission.RECEIVE_BOOT_COMPLETED', + 'android.permission.RECORD_VIDEO', + 'android.permission.REORDER_TASKS', + 'android.permission.REQUEST_INSTALL_PACKAGES', + 'android.permission.RESTRICTED_VR_ACCESS', + 'android.permission.RUN_INSTRUMENTATION', + 'android.permission.SET_ALARM', + 'android.permission.SET_TIME_ZONE', + 'android.permission.SET_WALLPAPER', + 'android.permission.SET_WALLPAPER_HINTS', + 'android.permission.TRANSMIT_IR', + 'android.permission.USE_CREDENTIALS', + 'android.permission.USE_FINGERPRINT', + 'android.permission.VIBRATE', + 'android.permission.WAKE_LOCK', + 'android.permission.WRITE_SYNC_SETTINGS', + 'com.android.browser.permission.READ_HISTORY_BOOKMARKS', + 'com.android.browser.permission.WRITE_HISTORY_BOOKMARKS', + 'com.android.launcher.permission.INSTALL_SHORTCUT', + 'com.chrome.permission.DEVICE_EXTRAS', + 'com.google.android.apps.now.CURRENT_ACCOUNT_ACCESS', + 'com.google.android.c2dm.permission.RECEIVE', + 'com.google.android.providers.gsf.permission.READ_GSERVICES', + 'com.google.vr.vrcore.permission.VRCORE_INTERNAL', + 'com.sec.enterprise.knox.MDM_CONTENT_PROVIDER', + '*.permission.C2D_MESSAGE', + '*.permission.READ_WRITE_BOOKMARK_FOLDERS', + '*.TOS_ACKED', + ])) _SHELL_OUTPUT_SEPARATOR = '~X~' -_PERMISSIONS_EXCEPTION_RE = re.compile( - r'java\.lang\.\w+Exception: .*$', re.MULTILINE) +_PERMISSIONS_EXCEPTION_RE = re.compile(r'java\.lang\.\w+Exception: .*$', + re.MULTILINE) _CURRENT_FOCUS_CRASH_RE = re.compile( r'\s*mCurrentFocus.*Application (Error|Not Responding): (\S+)}') _GETPROP_RE = re.compile(r'\[(.*?)\]: \[(.*?)\]') +_VERSION_CODE_SDK_RE = re.compile( + r'\s*versionCode=(\d+).*minSdk=(\d+).*targetSdk=(.*)\s*') # Regex to parse the long (-l) output of 'ls' command, c.f. # https://github.com/landley/toybox/blob/master/toys/posix/ls.c#L446 +# yapf: disable _LONG_LS_OUTPUT_RE = re.compile( r'(?P<st_mode>[\w-]{10})\s+' # File permissions r'(?:(?P<st_nlink>\d+)\s+)?' # Number of links (optional) @@ -187,44 +193,60 @@ _LONG_LS_OUTPUT_RE = re.compile( r'(?: -> (?P<symbolic_link_to>.+))?' # Symbolic link (optional) r'$' # End of string ) +# yapf: enable + _LS_DATE_FORMAT = '%Y-%m-%d %H:%M' _FILE_MODE_RE = re.compile(r'[dbclps-](?:[r-][w-][xSs-]){2}[r-][w-][xTt-]$') _FILE_MODE_KIND = { - 'd': stat.S_IFDIR, 'b': stat.S_IFBLK, 'c': stat.S_IFCHR, - 'l': stat.S_IFLNK, 'p': stat.S_IFIFO, 's': stat.S_IFSOCK, - '-': stat.S_IFREG} + 'd': stat.S_IFDIR, + 'b': stat.S_IFBLK, + 'c': stat.S_IFCHR, + 'l': stat.S_IFLNK, + 'p': stat.S_IFIFO, + 's': stat.S_IFSOCK, + '-': stat.S_IFREG +} _FILE_MODE_PERMS = [ - stat.S_IRUSR, stat.S_IWUSR, stat.S_IXUSR, - stat.S_IRGRP, stat.S_IWGRP, stat.S_IXGRP, - stat.S_IROTH, stat.S_IWOTH, stat.S_IXOTH, + stat.S_IRUSR, + stat.S_IWUSR, + stat.S_IXUSR, + stat.S_IRGRP, + stat.S_IWGRP, + stat.S_IXGRP, + stat.S_IROTH, + stat.S_IWOTH, + stat.S_IXOTH, ] _FILE_MODE_SPECIAL = [ ('s', stat.S_ISUID), ('s', stat.S_ISGID), ('t', stat.S_ISVTX), ] -_PS_COLUMNS = { - 'pid': 1, - 'ppid': 2, - 'name': -1 -} -_SELINUX_MODE = { - 'enforcing': True, - 'permissive': False, - 'disabled': None -} +_PS_COLUMNS = {'pid': 1, 'ppid': 2, 'name': -1} +_SELINUX_MODE = {'enforcing': True, 'permissive': False, 'disabled': None} # Some devices require different logic for checking if root is necessary _SPECIAL_ROOT_DEVICE_LIST = [ - 'marlin', # Pixel XL - 'sailfish', # Pixel - 'taimen', # Pixel 2 XL - 'vega', # Lenovo Mirage Solo - 'walleye', # Pixel 2 - 'crosshatch', # Pixel 3 XL - 'blueline', # Pixel 3 + 'marlin', # Pixel XL + 'sailfish', # Pixel + 'taimen', # Pixel 2 XL + 'vega', # Lenovo Mirage Solo + 'walleye', # Pixel 2 + 'crosshatch', # Pixel 3 XL + 'blueline', # Pixel 3 + 'sargo', # Pixel 3a + 'bonito', # Pixel 3a XL + 'sdk_goog3_x86', # Crow emulator +] +_SPECIAL_ROOT_DEVICE_LIST += [ + 'aosp_%s' % _d for _d in _SPECIAL_ROOT_DEVICE_LIST +] + +# Somce devices are slow/timeout when using default install. +# Devices listed here will perform no_streaming app installation. +_NO_STREAMING_DEVICE_LIST = [ + 'flounder', # Nexus 9 + 'volantis', # Another product name for Nexus 9 ] -_SPECIAL_ROOT_DEVICE_LIST += ['aosp_%s' % _d for _d in - _SPECIAL_ROOT_DEVICE_LIST] _IMEI_RE = re.compile(r' Device ID = (.+)$') # The following regex is used to match result parcels like: @@ -236,8 +258,6 @@ Result: Parcel( """ _PARCEL_RESULT_RE = re.compile( r'0x[0-9a-f]{8}\: (?:[0-9a-f]{8}\s+){1,4}\'(.{16})\'') -_EBUSY_RE = re.compile( - r'mkdir failed for ([^,]*), Device or resource busy') # http://bit.ly/2WLZhUF added a timeout to adb wait-for-device. We sometimes # want to wait longer than the implicit call within adb root allows. @@ -245,8 +265,7 @@ _WAIT_FOR_DEVICE_TIMEOUT_STR = 'timeout expired while waiting for device' _WEBVIEW_SYSUPDATE_CURRENT_PKG_RE = re.compile( r'Current WebView package.*:.*\(([a-z.]*),') -_WEBVIEW_SYSUPDATE_NULL_PKG_RE = re.compile( - r'Current WebView package is null') +_WEBVIEW_SYSUPDATE_NULL_PKG_RE = re.compile(r'Current WebView package is null') _WEBVIEW_SYSUPDATE_FALLBACK_LOGIC_RE = re.compile( r'Fallback logic enabled: (true|false)') _WEBVIEW_SYSUPDATE_PACKAGE_INSTALLED_RE = re.compile( @@ -262,8 +281,7 @@ PS_COLUMNS = ('name', 'pid', 'ppid') ProcessInfo = collections.namedtuple('ProcessInfo', PS_COLUMNS) -@decorators.WithExplicitTimeoutAndRetries( - _DEFAULT_TIMEOUT, _DEFAULT_RETRIES) +@decorators.WithExplicitTimeoutAndRetries(_DEFAULT_TIMEOUT, _DEFAULT_RETRIES) def GetAVDs(): """Returns a list of Android Virtual Devices. @@ -271,9 +289,10 @@ def GetAVDs(): A list containing the configured AVDs. """ lines = cmd_helper.GetCmdOutput([ - os.path.join(devil_env.config.LocalPath('android_sdk'), - 'tools', 'android'), - 'list', 'avd']).splitlines() + os.path.join( + devil_env.config.LocalPath('android_sdk'), 'tools', 'android'), + 'list', 'avd' + ]).splitlines() avds = [] for line in lines: if 'Name:' not in line: @@ -284,14 +303,14 @@ def GetAVDs(): return avds -@decorators.WithExplicitTimeoutAndRetries( - _DEFAULT_TIMEOUT, _DEFAULT_RETRIES) +@decorators.WithExplicitTimeoutAndRetries(_DEFAULT_TIMEOUT, _DEFAULT_RETRIES) def RestartServer(): """Restarts the adb server. Raises: CommandFailedError if we fail to kill or restart the server. """ + def adb_killed(): return not adb_wrapper.AdbWrapper.IsServerOnline() @@ -300,7 +319,8 @@ def RestartServer(): adb_wrapper.AdbWrapper.KillServer() if not timeout_retry.WaitFor(adb_killed, wait_period=1, max_tries=5): - # TODO(perezju): raise an exception after fixng http://crbug.com/442319 + # TODO(crbug.com/442319): Switch this to raise an exception if we + # figure out why sometimes not all adb servers on bots get killed. logger.warning('Failed to kill adb server') adb_wrapper.AdbWrapper.StartServer() if not timeout_retry.WaitFor(adb_started, wait_period=1, max_tries=5): @@ -356,12 +376,62 @@ def _FormatPartialOutputError(output): return '\n'.join(message) +_PushableComponents = collections.namedtuple('_PushableComponents', + ('host', 'device', 'collapse')) + + +def _IterPushableComponents(host_path, device_path): + """Yields a sequence of paths that can be pushed directly via adb push. + + `adb push` doesn't currently handle pushing directories that contain + symlinks: https://bit.ly/2pMBlW5 + + To circumvent this issue, we get the smallest set of files and/or + directories that can be pushed without attempting to push a directory + that contains a symlink. + + This function does so by recursing through |host_path|. Each call + yields 3-tuples that include the smallest set of (host, device) path pairs + that can be passed to adb push and a bool indicating whether the parent + directory can be pushed -- i.e., if True, the host path is neither a + symlink nor a directory that contains a symlink. + + Args: + host_path: an absolute path of a file or directory on the host + device_path: an absolute path of a file or directory on the device + Yields: + 3-tuples containing + host (str): the host path, with symlinks dereferenced + device (str): the device path + collapse (bool): whether this entity permits its parent to be pushed + in its entirety. (Parents need permission from all child entities + in order to be pushed in their entirety.) + """ + if os.path.isfile(host_path): + yield _PushableComponents( + os.path.realpath(host_path), device_path, not os.path.islink(host_path)) + else: + components = [] + for child in os.listdir(host_path): + components.extend( + _IterPushableComponents( + os.path.join(host_path, child), posixpath.join( + device_path, child))) + + if all(c.collapse for c in components): + yield _PushableComponents( + os.path.realpath(host_path), device_path, + not os.path.islink(host_path)) + else: + for c in components: + yield c + + class DeviceUtils(object): _MAX_ADB_COMMAND_LENGTH = 512 _MAX_ADB_OUTPUT_LENGTH = 32768 - _LAUNCHER_FOCUSED_RE = re.compile( - r'\s*mCurrentFocus.*(Launcher|launcher).*') + _LAUNCHER_FOCUSED_RE = re.compile(r'\s*mCurrentFocus.*(Launcher|launcher).*') _VALID_SHELL_VARIABLE = re.compile('^[a-zA-Z_][a-zA-Z0-9_]*$') LOCAL_PROPERTIES_PATH = posixpath.join('/', 'data', 'local.prop') @@ -369,7 +439,9 @@ class DeviceUtils(object): # Property in /data/local.prop that controls Java assertions. JAVA_ASSERT_PROPERTY = 'dalvik.vm.enableassertions' - def __init__(self, device, enable_device_files_cache=False, + def __init__(self, + device, + enable_device_files_cache=False, default_timeout=_DEFAULT_TIMEOUT, default_retries=_DEFAULT_RETRIES): """DeviceUtils constructor. @@ -489,7 +561,12 @@ class DeviceUtils(object): # 'eng' builds have root enabled by default and the adb session cannot # be unrooted. return True - if self.product_name in _SPECIAL_ROOT_DEVICE_LIST: + # Devices using the system-as-root partition layout appear to not have + # a /root directory. See http://bit.ly/37F34sx for more context. + if (self.build_system_root_image == 'true' + or self.build_version_sdk >= version_codes.Q + # This may be redundant with the checks above. + or self.product_name in _SPECIAL_ROOT_DEVICE_LIST): return self.GetProp('service.adb.root') == '1' self.RunShellCommand(['ls', '/root'], check_return=True) return True @@ -515,13 +592,21 @@ class DeviceUtils(object): """ if 'needs_su' not in self._cache: cmd = '%s && ! ls /root' % self._Su('ls /root') - if self.product_name in _SPECIAL_ROOT_DEVICE_LIST: + # Devices using the system-as-root partition layout appear to not have + # a /root directory. See http://bit.ly/37F34sx for more context. + if (self.build_system_root_image == 'true' + or self.build_version_sdk >= version_codes.Q + # This may be redundant with the checks above. + or self.product_name in _SPECIAL_ROOT_DEVICE_LIST): if self.HasRoot(): self._cache['needs_su'] = False return False cmd = 'which which && which su' try: - self.RunShellCommand(cmd, shell=True, check_return=True, + self.RunShellCommand( + cmd, + shell=True, + check_return=True, timeout=self._default_timeout if timeout is DEFAULT else timeout, retries=self._default_retries if retries is DEFAULT else retries) self._cache['needs_su'] = True @@ -529,7 +614,6 @@ class DeviceUtils(object): self._cache['needs_su'] = False return self._cache['needs_su'] - def _Su(self, command): if self.build_version_sdk >= version_codes.MARSHMALLOW: return 'su 0 %s' % command @@ -596,6 +680,9 @@ class DeviceUtils(object): def GetExternalStoragePath(self, timeout=None, retries=None): """Get the device's path to its SD card. + Note: this path is read-only by apps in R+. Use GetAppWritablePath() to + obtain a path writable by apps. + Args: timeout: timeout in seconds retries: number of retries @@ -614,6 +701,30 @@ class DeviceUtils(object): str(self)) return self._cache['external_storage'] + def GetAppWritablePath(self, timeout=None, retries=None): + """Get a path that on the device's SD card that apps can write. + + Args: + timeout: timeout in seconds + retries: number of retries + + Returns: + A app-writeable path on the device's SD card. + + Raises: + CommandFailedError if the external storage path could not be determined. + CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. + """ + if self.build_version_sdk >= version_codes.Q: + # On Q+ apps don't require permissions to access well-defined media + # locations like /sdcard/Download. On R+ the WRITE_EXTERNAL_STORAGE + # permission no longer provides access to the external storage root. See + # https://developer.android.com/preview/privacy/storage#permissions-target-11 + # So use /sdcard/Download for the app-writable path on those versions. + return posixpath.join(self.GetExternalStoragePath(), 'Download') + return self.GetExternalStoragePath() + @decorators.WithTimeoutAndRetriesFromInstance() def GetIMEI(self, timeout=None, retries=None): """Get the device's IMEI. @@ -633,7 +744,8 @@ class DeviceUtils(object): if self.build_version_sdk < 21: out = self.RunShellCommand(['dumpsys', 'iphonesubinfo'], - raw_output=True, check_return=True) + raw_output=True, + check_return=True) if out: match = re.search(_IMEI_RE, out) if match: @@ -656,6 +768,23 @@ class DeviceUtils(object): raise device_errors.CommandFailedError('Unable to fetch IMEI.') @decorators.WithTimeoutAndRetriesFromInstance() + def IsApplicationInstalled(self, package, timeout=None, retries=None): + """Determines whether a particular package is installed on the device. + + Args: + package: Name of the package. + + Returns: + True if the application is installed, False otherwise. + """ + # `pm list packages` allows matching substrings, but we want exact matches + # only. + matching_packages = self.RunShellCommand( + ['pm', 'list', 'packages', package], check_return=True) + desired_line = 'package:' + package + return desired_line in matching_packages + + @decorators.WithTimeoutAndRetriesFromInstance() def GetApplicationPaths(self, package, timeout=None, retries=None): """Get the paths of the installed apks on the device for the given package. @@ -685,8 +814,8 @@ class DeviceUtils(object): # TODO(jbudorick): Check if this is fixed as new Android versions are # released to put an upper bound on this. should_check_return = (self.build_version_sdk < version_codes.LOLLIPOP) - output = self.RunShellCommand( - ['pm', 'path', package], check_return=should_check_return) + output = self.RunShellCommand(['pm', 'path', package], + check_return=should_check_return) apks = [] bad_output = False for line in output: @@ -718,8 +847,8 @@ class DeviceUtils(object): A string with the version name or None if the package is not found on the device. """ - output = self.RunShellCommand( - ['dumpsys', 'package', package], check_return=True) + output = self.RunShellCommand(['dumpsys', 'package', package], + check_return=True) if not output: return None for line in output: @@ -730,6 +859,35 @@ class DeviceUtils(object): 'Version name for %s not found on dumpsys output' % package, str(self)) @decorators.WithTimeoutAndRetriesFromInstance() + def GetApplicationTargetSdk(self, package, timeout=None, retries=None): + """Get the targetSdkVersion of a package installed on the device. + + Args: + package: Name of the package. + + Returns: + A string with the targetSdkVersion or None if the package is not found on + the device. Note: this cannot always be cast to an integer. If this + application targets a pre-release SDK, this returns the version codename + instead (ex. "R"). + """ + if not self.IsApplicationInstalled(package): + return None + lines = self._GetDumpsysOutput(['package', package], 'targetSdk=') + for line in lines: + m = _VERSION_CODE_SDK_RE.match(line) + if m: + value = m.group(3) + # 10000 is the code used by Android for a pre-finalized SDK. + if value == '10000': + return self.GetProp('ro.build.version.codename', cache=True) + else: + return value + raise device_errors.CommandFailedError( + 'targetSdkVersion for %s not found on dumpsys output' % package, + str(self)) + + @decorators.WithTimeoutAndRetriesFromInstance() def GetPackageArchitecture(self, package, timeout=None, retries=None): """Get the architecture of a package installed on the device. @@ -758,6 +916,9 @@ class DeviceUtils(object): CommandFailedError if the package's data directory can't be found, whether because it's not installed or otherwise. """ + if not self.IsApplicationInstalled(package): + raise device_errors.CommandFailedError('%s is not installed' % package, + str(self)) output = self._RunPipedShellCommand( 'pm dump %s | grep dataDir=' % cmd_helper.SingleQuote(package)) for line in output: @@ -765,11 +926,14 @@ class DeviceUtils(object): if dataDir: return dataDir raise device_errors.CommandFailedError( - 'Could not find data directory for %s', package) + 'Could not find data directory for %s' % package, str(self)) @decorators.WithTimeoutAndRetriesFromInstance() - def GetSecurityContextForPackage(self, package, encrypted=False, timeout=None, - retries=None): + def GetSecurityContextForPackage(self, + package, + encrypted=False, + timeout=None, + retries=None): """Gets the SELinux security context for the given package. Args: @@ -782,7 +946,8 @@ class DeviceUtils(object): """ directory = '/data/user_de/0/' if encrypted else '/data/data/' for line in self.RunShellCommand(['ls', '-Z', directory], - as_root=True, check_return=True): + as_root=True, + check_return=True): split_line = line.split() # ls -Z output differs between Android versions, but the package is # always last and the context always starts with "u:object" @@ -792,7 +957,7 @@ class DeviceUtils(object): return column return None - def TakeBugReport(self, path, timeout=60*5, retries=None): + def TakeBugReport(self, path, timeout=60 * 5, retries=None): """Takes a bug report and dumps it to the specified path. This doesn't use adb's bugreport option since its behavior is dependent on @@ -812,15 +977,23 @@ class DeviceUtils(object): self.PullFile(device_tmp_file.name, path) @decorators.WithTimeoutAndRetriesFromInstance() - def WaitUntilFullyBooted(self, wifi=False, timeout=None, retries=None): + def WaitUntilFullyBooted(self, + wifi=False, + decrypt=False, + timeout=None, + retries=None): """Wait for the device to fully boot. This means waiting for the device to boot, the package manager to be - available, and the SD card to be ready. It can optionally mean waiting - for wifi to come up, too. + available, and the SD card to be ready. + It can optionally wait the following: + - Wait for wifi to come up. + - Wait for full-disk decryption to complete. Args: wifi: A boolean indicating if we should wait for wifi to come up or not. + decrypt: A boolean indicating if we should wait for full-disk decryption + to complete. timeout: timeout in seconds retries: number of retries @@ -829,10 +1002,11 @@ class DeviceUtils(object): CommandTimeoutError if one of the component waits times out. DeviceUnreachableError if the device becomes unresponsive. """ + def sd_card_ready(): try: - self.RunShellCommand(['test', '-d', self.GetExternalStoragePath()], - check_return=True) + self.RunShellCommand( + ['test', '-d', self.GetExternalStoragePath()], check_return=True) return True except device_errors.AdbCommandFailedError: return False @@ -853,24 +1027,49 @@ class DeviceUtils(object): return 'Wi-Fi is enabled' in self.RunShellCommand(['dumpsys', 'wifi'], check_return=False) + def decryption_completed(): + try: + decrypt = self.GetProp('vold.decrypt', cache=False) + # The prop "void.decrypt" will only be set when the device uses + # full-disk encryption (FDE). + # Return true when: + # - The prop is empty, which means the device is unencrypted or uses + # file-based encryption (FBE). + # - or the prop has value "trigger_restart_framework", which means + # the decription is finished. + return decrypt == '' or decrypt == 'trigger_restart_framework' + except device_errors.CommandFailedError: + return False + self.adb.WaitForDevice() timeout_retry.WaitFor(sd_card_ready) timeout_retry.WaitFor(pm_ready) timeout_retry.WaitFor(boot_completed) if wifi: timeout_retry.WaitFor(wifi_enabled) + if decrypt: + timeout_retry.WaitFor(decryption_completed) REBOOT_DEFAULT_TIMEOUT = 10 * _DEFAULT_TIMEOUT @decorators.WithTimeoutAndRetriesFromInstance( min_default_timeout=REBOOT_DEFAULT_TIMEOUT) - def Reboot(self, block=True, wifi=False, timeout=None, retries=None): + def Reboot(self, + block=True, + wifi=False, + decrypt=False, + timeout=None, + retries=None): """Reboot the device. Args: block: A boolean indicating if we should wait for the reboot to complete. wifi: A boolean indicating if we should wait for wifi to be enabled after - the reboot. The option has no effect unless |block| is also True. + the reboot. + The option has no effect unless |block| is also True. + decrypt: A boolean indicating if we should wait for full-disk decryption + to complete after the reboot. + The option has no effect unless |block| is also True. timeout: timeout in seconds retries: number of retries @@ -878,6 +1077,7 @@ class DeviceUtils(object): CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ + def device_offline(): return not self.IsOnline() @@ -885,14 +1085,23 @@ class DeviceUtils(object): self.ClearCache() timeout_retry.WaitFor(device_offline, wait_period=1) if block: - self.WaitUntilFullyBooted(wifi=wifi) + self.WaitUntilFullyBooted(wifi=wifi, decrypt=decrypt) INSTALL_DEFAULT_TIMEOUT = 8 * _DEFAULT_TIMEOUT + MODULES_SRC_DIRECTORY_PATH = '/data/local/tmp/modules' @decorators.WithTimeoutAndRetriesFromInstance( min_default_timeout=INSTALL_DEFAULT_TIMEOUT) - def Install(self, apk, allow_downgrade=False, reinstall=False, - permissions=None, timeout=None, retries=None, modules=None): + def Install(self, + apk, + allow_downgrade=False, + reinstall=False, + permissions=None, + timeout=None, + retries=None, + modules=None, + fake_modules=None, + additional_locales=None): """Install an APK or app bundle. Noop if an identical APK is already installed. If installing a bundle, the @@ -911,21 +1120,93 @@ class DeviceUtils(object): retries: number of retries modules: An iterable containing specific bundle modules to install. Error if set and |apk| points to an APK instead of a bundle. + fake_modules: An iterable containing specific bundle modules that should + have their apks copied to |MODULES_SRC_DIRECTORY_PATH| subdirectory + rather than installed. Thus the app can emulate SplitCompat while + running. This should not have any overlap with |modules|. + additional_locales: An iterable with additional locales to install for a + bundle. Raises: CommandFailedError if the installation fails. CommandTimeoutError if the installation times out. DeviceUnreachableError on missing device. """ - self._InstallInternal(apk, None, allow_downgrade=allow_downgrade, - reinstall=reinstall, permissions=permissions, - modules=modules) + apk = apk_helper.ToHelper(apk) + modules_set = set(modules or []) + fake_modules_set = set(fake_modules or []) + assert modules_set.isdisjoint(fake_modules_set), ( + 'These modules overlap: %s' % (modules_set & fake_modules_set)) + all_modules = modules_set | fake_modules_set + package_name = apk.GetPackageName() + + with apk.GetApkPaths(self, + modules=all_modules, + additional_locales=additional_locales) as apk_paths: + if apk.SupportsSplits(): + fake_apk_paths = self._GetFakeInstallPaths(apk_paths, fake_modules) + self._FakeInstall(fake_apk_paths, fake_modules, package_name) + apk_paths_to_install = [p for p in apk_paths if p not in fake_apk_paths] + else: + apk_paths_to_install = apk_paths + self._InstallInternal( + apk, + apk_paths_to_install, + allow_downgrade=allow_downgrade, + reinstall=reinstall, + permissions=permissions) + + @staticmethod + def _GetFakeInstallPaths(apk_paths, fake_modules): + def IsFakeModulePath(path): + filename = os.path.basename(path) + return any(filename.startswith(f + '-') for f in fake_modules) + + if not fake_modules: + return set() + return set(p for p in apk_paths if IsFakeModulePath(p)) + + def _FakeInstall(self, fake_apk_paths, fake_modules, package_name): + with tempfile_ext.NamedTemporaryDirectory() as modules_dir: + device_dir = posixpath.join(self.MODULES_SRC_DIRECTORY_PATH, package_name) + if not fake_modules: + # Push empty module dir to clear device dir and update the cache. + self.PushChangedFiles([(modules_dir, device_dir)], + delete_device_stale=True) + return + + still_need_master = set(fake_modules) + for path in fake_apk_paths: + filename = os.path.basename(path) + # Example names: base-en.apk, test_dummy-master.apk. + module_name, suffix = filename.split('-', 1) + if 'master' in suffix: + assert module_name in still_need_master, ( + 'Duplicate master apk file for %s' % module_name) + still_need_master.remove(module_name) + new_filename = '%s.apk' % module_name + else: + # |suffix| includes .apk extension. + new_filename = '%s.config.%s' % (module_name, suffix) + new_path = os.path.join(modules_dir, new_filename) + os.rename(path, new_path) + + assert not still_need_master, ( + 'Missing master apk file for %s' % still_need_master) + self.PushChangedFiles([(modules_dir, device_dir)], + delete_device_stale=True) @decorators.WithTimeoutAndRetriesFromInstance( min_default_timeout=INSTALL_DEFAULT_TIMEOUT) - def InstallSplitApk(self, base_apk, split_apks, allow_downgrade=False, - reinstall=False, allow_cached_props=False, - permissions=None, timeout=None, retries=None): + def InstallSplitApk(self, + base_apk, + split_apks, + allow_downgrade=False, + reinstall=False, + allow_cached_props=False, + permissions=None, + timeout=None, + retries=None): """Install a split APK. Noop if all of the APK splits are already installed. @@ -948,73 +1229,57 @@ class DeviceUtils(object): DeviceUnreachableError on missing device. DeviceVersionError if device SDK is less than Android L. """ - self._InstallInternal(base_apk, split_apks, reinstall=reinstall, - allow_cached_props=allow_cached_props, - permissions=permissions, - allow_downgrade=allow_downgrade) - - def _InstallInternal(self, base_apk, split_apks, allow_downgrade=False, - reinstall=False, allow_cached_props=False, - permissions=None, modules=None): - base_apk = apk_helper.ToHelper(base_apk) - if base_apk.is_bundle: - if split_apks: - raise device_errors.CommandFailedError( - 'Attempted to install a bundle {} while specifying split apks' - .format(base_apk)) - if allow_downgrade: - logging.warning('Installation of a bundle requested with ' - 'allow_downgrade=False. This is not possible with ' - 'bundletools, no downgrading is possible. This ' - 'flag will be ignored and installation will proceed.') - # |allow_cached_props| is unused and ignored for bundles. - self._InstallBundleInternal(base_apk, permissions, modules) - return - - if modules: - raise device_errors.CommandFailedError( - 'Attempted to specify modules to install when providing an APK') - - if split_apks: + apk = apk_helper.ToSplitHelper(base_apk, split_apks) + with apk.GetApkPaths( + self, allow_cached_props=allow_cached_props) as apk_paths: + self._InstallInternal( + apk, + apk_paths, + reinstall=reinstall, + permissions=permissions, + allow_downgrade=allow_downgrade) + + def _InstallInternal(self, + apk, + apk_paths, + allow_downgrade=False, + reinstall=False, + permissions=None): + if not apk_paths: + raise device_errors.CommandFailedError('Did not get any APKs to install') + + if len(apk_paths) > 1: self._CheckSdkLevel(version_codes.LOLLIPOP) - all_apks = [base_apk.path] - if split_apks: - all_apks += split_select.SelectSplits( - self, base_apk.path, split_apks, allow_cached_props=allow_cached_props) - if len(all_apks) == 1: - logger.warning('split-select did not select any from %s', split_apks) - - missing_apks = [apk for apk in all_apks if not os.path.exists(apk)] + missing_apks = [a for a in apk_paths if not os.path.exists(a)] if missing_apks: raise device_errors.CommandFailedError( - 'Attempted to install non-existent apks: %s' - % pprint.pformat(missing_apks)) + 'Attempted to install non-existent apks: %s' % + pprint.pformat(missing_apks)) - package_name = base_apk.GetPackageName() + package_name = apk.GetPackageName() device_apk_paths = self._GetApplicationPathsInternal(package_name) - apks_to_install = None host_checksums = None if not device_apk_paths: - apks_to_install = all_apks - elif len(device_apk_paths) > 1 and not split_apks: + apks_to_install = apk_paths + elif len(device_apk_paths) > 1 and len(apk_paths) == 1: logger.warning( 'Installing non-split APK when split APK was previously installed') - apks_to_install = all_apks - elif len(device_apk_paths) == 1 and split_apks: + apks_to_install = apk_paths + elif len(device_apk_paths) == 1 and len(apk_paths) > 1: logger.warning( 'Installing split APK when non-split APK was previously installed') - apks_to_install = all_apks + apks_to_install = apk_paths else: try: - apks_to_install, host_checksums = ( - self._ComputeStaleApks(package_name, all_apks)) - except EnvironmentError as e: + apks_to_install, host_checksums = (self._ComputeStaleApks( + package_name, apk_paths)) + except device_errors.CommandFailedError as e: logger.warning('Error calculating md5: %s', e) - apks_to_install, host_checksums = all_apks, None + apks_to_install, host_checksums = apk_paths, None if apks_to_install and not reinstall: - apks_to_install = all_apks + apks_to_install = apk_paths if device_apk_paths and apks_to_install and not reinstall: self.Uninstall(package_name) @@ -1023,14 +1288,23 @@ class DeviceUtils(object): # Assume that we won't know the resulting device state. self._cache['package_apk_paths'].pop(package_name, 0) self._cache['package_apk_checksums'].pop(package_name, 0) - if split_apks: - partial = package_name if len(apks_to_install) < len(all_apks) else None + partial = package_name if len(apks_to_install) < len(apk_paths) else None + streaming = None + if self.product_name in _NO_STREAMING_DEVICE_LIST: + streaming = False + if len(apks_to_install) > 1 or partial: self.adb.InstallMultiple( - apks_to_install, partial=partial, reinstall=reinstall, + apks_to_install, + partial=partial, + reinstall=reinstall, + streaming=streaming, allow_downgrade=allow_downgrade) else: self.adb.Install( - base_apk.path, reinstall=reinstall, allow_downgrade=allow_downgrade) + apks_to_install[0], + reinstall=reinstall, + streaming=streaming, + allow_downgrade=allow_downgrade) else: # Running adb install terminates running instances of the app, so to be # consistent, we explicitly terminate it when skipping the install. @@ -1038,26 +1312,12 @@ class DeviceUtils(object): if (permissions is None and self.build_version_sdk >= version_codes.MARSHMALLOW): - permissions = base_apk.GetPermissions() + permissions = apk.GetPermissions() self.GrantPermissions(package_name, permissions) # Upon success, we know the device checksums, but not their paths. if host_checksums is not None: self._cache['package_apk_checksums'][package_name] = host_checksums - def _InstallBundleInternal(self, bundle, permissions, modules): - cmd = [bundle.path, 'install', '--device', self.serial] - if modules: - for m in modules: - cmd.extend(['-m', m]) - status = cmd_helper.RunCmd(cmd) - if status != 0: - raise device_errors.CommandFailedError('Cound not install {}'.format( - bundle.path)) - if (permissions is None - and self.build_version_sdk >= version_codes.MARSHMALLOW): - permissions = bundle.GetPermissions() - self.GrantPermissions(bundle.GetPackageName(), permissions) - @decorators.WithTimeoutAndRetriesFromInstance() def Uninstall(self, package_name, keep_data=False, timeout=None, retries=None): @@ -1092,12 +1352,21 @@ class DeviceUtils(object): raise device_errors.DeviceVersionError( ('Requires SDK level %s, device is SDK level %s' % (required_sdk_level, self.build_version_sdk)), - device_serial=self.serial) + device_serial=self.serial) @decorators.WithTimeoutAndRetriesFromInstance() - def RunShellCommand(self, cmd, shell=False, check_return=False, cwd=None, - env=None, run_as=None, as_root=False, single_line=False, - large_output=False, raw_output=False, timeout=None, + def RunShellCommand(self, + cmd, + shell=False, + check_return=False, + cwd=None, + env=None, + run_as=None, + as_root=False, + single_line=False, + large_output=False, + raw_output=False, + timeout=None, retries=None): """Run an ADB shell command. @@ -1120,8 +1389,8 @@ class DeviceUtils(object): This behaviour is consistent with that of command runners in cmd_helper as well as Python's own subprocess.Popen. - TODO(perezju) Change the default of |check_return| to True when callers - have switched to the new behaviour. + TODO(crbug.com/1029769) Change the default of |check_return| to True when + callers have switched to the new behaviour. Args: cmd: A sequence containing the command to run and its arguments, or a @@ -1157,6 +1426,7 @@ class DeviceUtils(object): CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ + def env_quote(key, value): if not DeviceUtils._VALID_SHELL_VARIABLE.match(key): raise KeyError('Invalid shell variable name %r' % key) @@ -1181,8 +1451,8 @@ class DeviceUtils(object): else: with device_temp_file.DeviceTempFile(self.adb, suffix='.sh') as script: self._WriteFileWithPush(script.name, cmd) - logger.info('Large shell command will be run from file: %s ...', - cmd[:self._MAX_ADB_COMMAND_LENGTH]) + logger.debug('Large shell command will be run from file: %s ...', + cmd[:self._MAX_ADB_COMMAND_LENGTH]) return handle_check_return('sh %s' % script.name_quoted) def handle_large_output(cmd, large_output_mode): @@ -1213,6 +1483,7 @@ class DeviceUtils(object): if isinstance(cmd, basestring): if not shell: + # TODO(crbug.com/1029769): Make this an error instead. logger.warning( 'The command to run should preferably be passed as a sequence of' ' args. If shell features are needed (pipes, wildcards, variables)' @@ -1259,22 +1530,27 @@ class DeviceUtils(object): if not pipestatus_line.startswith(PIPESTATUS_LEADER): logger.error('Pipe exit statuses of shell script missing.') raise device_errors.AdbShellCommandFailedError( - script, output, status=None, - device_serial=self.serial) + script, output, status=None, device_serial=self.serial) output = output[:-1] statuses = [ - int(s) for s in pipestatus_line[len(PIPESTATUS_LEADER):].split()] + int(s) for s in pipestatus_line[len(PIPESTATUS_LEADER):].split() + ] if any(statuses): raise device_errors.AdbShellCommandFailedError( - script, output, status=statuses, - device_serial=self.serial) + script, output, status=statuses, device_serial=self.serial) return output @decorators.WithTimeoutAndRetriesFromInstance() - def KillAll(self, process_name, exact=False, signum=device_signal.SIGKILL, - as_root=False, blocking=False, quiet=False, - timeout=None, retries=None): + def KillAll(self, + process_name, + exact=False, + signum=device_signal.SIGKILL, + as_root=False, + blocking=False, + quiet=False, + timeout=None, + retries=None): """Kill all processes with the given name on the device. Args: @@ -1312,8 +1588,8 @@ class DeviceUtils(object): 'No processes matching %r (exact=%r)' % (process_name, exact), str(self)) - logger.info( - 'KillAll(%r, ...) attempting to kill the following:', process_name) + logger.info('KillAll(%r, ...) attempting to kill the following:', + process_name) for p in processes: logger.info(' %05d %s', p.pid, p.name) @@ -1331,8 +1607,13 @@ class DeviceUtils(object): return len(pids) @decorators.WithTimeoutAndRetriesFromInstance() - def StartActivity(self, intent_obj, blocking=False, trace_file_name=None, - force_stop=False, timeout=None, retries=None): + def StartActivity(self, + intent_obj, + blocking=False, + trace_file_name=None, + force_stop=False, + timeout=None, + retries=None): """Start package's activity on the device. Args: @@ -1392,8 +1673,13 @@ class DeviceUtils(object): raise device_errors.CommandFailedError(line, str(self)) @decorators.WithTimeoutAndRetriesFromInstance() - def StartInstrumentation(self, component, finish=True, raw=False, - extras=None, timeout=None, retries=None): + def StartInstrumentation(self, + component, + finish=True, + raw=False, + extras=None, + timeout=None, + retries=None): if extras is None: extras = {} @@ -1411,8 +1697,8 @@ class DeviceUtils(object): package = component.split('/')[0] shell_snippet = 'p=%s;%s' % (package, cmd_helper.ShrinkToSnippet(cmd, 'p', package)) - return self.RunShellCommand(shell_snippet, shell=True, check_return=True, - large_output=True) + return self.RunShellCommand( + shell_snippet, shell=True, check_return=True, large_output=True) @decorators.WithTimeoutAndRetriesFromInstance() def BroadcastIntent(self, intent_obj, timeout=None, retries=None): @@ -1445,9 +1731,11 @@ class DeviceUtils(object): CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ + def is_launcher_focused(): output = self.RunShellCommand(['dumpsys', 'window', 'windows'], - check_return=True, large_output=True) + check_return=True, + large_output=True) return any(self._LAUNCHER_FOCUSED_RE.match(l) for l in output) def dismiss_popups(): @@ -1462,8 +1750,9 @@ class DeviceUtils(object): return self.StartActivity( - intent.Intent(action='android.intent.action.MAIN', - category='android.intent.category.HOME'), + intent.Intent( + action='android.intent.action.MAIN', + category='android.intent.category.HOME'), blocking=True) if not is_launcher_focused(): @@ -1486,8 +1775,11 @@ class DeviceUtils(object): self.RunShellCommand(['am', 'force-stop', package], check_return=True) @decorators.WithTimeoutAndRetriesFromInstance() - def ClearApplicationState( - self, package, permissions=None, timeout=None, retries=None): + def ClearApplicationState(self, + package, + permissions=None, + timeout=None, + retries=None): """Clear all state for the given package. Args: @@ -1523,15 +1815,18 @@ class DeviceUtils(object): CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ - self.RunShellCommand(['input', 'keyevent', format(keycode, 'd')], - check_return=True) + self.RunShellCommand( + ['input', 'keyevent', format(keycode, 'd')], check_return=True) PUSH_CHANGED_FILES_DEFAULT_TIMEOUT = 10 * _DEFAULT_TIMEOUT @decorators.WithTimeoutAndRetriesFromInstance( min_default_timeout=PUSH_CHANGED_FILES_DEFAULT_TIMEOUT) - def PushChangedFiles(self, host_device_tuples, timeout=None, - retries=None, delete_device_stale=False): + def PushChangedFiles(self, + host_device_tuples, + delete_device_stale=False, + timeout=None, + retries=None): """Push files to the device, skipping files that don't need updating. When a directory is pushed, it is traversed recursively on the host and @@ -1544,194 +1839,208 @@ class DeviceUtils(object): |host_path| is an absolute path of a file or directory on the host that should be minimially pushed to the device, and |device_path| is an absolute path of the destination on the device. + delete_device_stale: option to delete stale files on device timeout: timeout in seconds retries: number of retries - delete_device_stale: option to delete stale files on device Raises: CommandFailedError on failure. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ + # TODO(crbug.com/1005504): Experiment with this on physical devices after + # upgrading devil's default adb beyond 1.0.39. + # TODO(crbug.com/1020716): disabled as can result in extra directory. + enable_push_sync = False - all_changed_files = [] - all_stale_files = [] - missing_dirs = set() - cache_commit_funcs = [] - for h, d in host_device_tuples: - assert os.path.isabs(h) and posixpath.isabs(d) - h = os.path.realpath(h) - changed_files, up_to_date_files, stale_files, cache_commit_func = ( - self._GetChangedAndStaleFiles(h, d, delete_device_stale)) - all_changed_files += changed_files - all_stale_files += stale_files - cache_commit_funcs.append(cache_commit_func) - if changed_files and not up_to_date_files and not stale_files: - if os.path.isdir(h): - missing_dirs.add(d) - else: - missing_dirs.add(posixpath.dirname(d)) + if enable_push_sync: + try: + self._PushChangedFilesSync(host_device_tuples) + return + except device_errors.AdbVersionError as e: + # If we don't meet the adb requirements, fall back to the previous + # sync-unaware implementation. + logging.warning(str(e)) - if delete_device_stale and all_stale_files: - self.RemovePath(all_stale_files, force=True, recursive=True) + changed_files, missing_dirs, cache_commit_func = (self._GetChangedFiles( + host_device_tuples, delete_device_stale)) - if all_changed_files: + if changed_files: if missing_dirs: - try: - self.RunShellCommand(['mkdir', '-p'] + list(missing_dirs), - check_return=True) - except device_errors.AdbShellCommandFailedError as e: - # TODO(crbug.com/739899): This is attempting to diagnose flaky EBUSY - # errors that have been popping up in single-device scenarios. - # Remove it once we've figured out what's causing them and how best - # to handle them. - m = _EBUSY_RE.search(e.output) - if m: - logging.error( - 'Hit EBUSY while attempting to make missing directories.') - logging.error('lsof output:') - # Don't check for return below since grep exits with a non-zero when - # no match is found. - for l in self.RunShellCommand( - 'lsof | grep %s' % cmd_helper.SingleQuote(m.group(1)), - check_return=False): - logging.error(' %s', l) - raise - self._PushFilesImpl(host_device_tuples, all_changed_files) - for func in cache_commit_funcs: - func() + self.RunShellCommand(['mkdir', '-p'] + list(missing_dirs), + check_return=True) + self._PushFilesImpl(host_device_tuples, changed_files) + cache_commit_func() + + def _PushChangedFilesSync(self, host_device_tuples): + """Push changed files via `adb sync`. + + Args: + host_device_tuples: Same as PushChangedFiles. + """ + for h, d in host_device_tuples: + for ph, pd, _ in _IterPushableComponents(h, d): + self.adb.Push(ph, pd, sync=True) + - def _GetChangedAndStaleFiles(self, host_path, device_path, track_stale=False): - """Get files to push and delete + def _GetDeviceNodes(self, paths): + """Get the set of all files and directories on the device contained within + the provided list of paths, without recursively expanding directories. Args: - host_path: an absolute path of a file or directory on the host - device_path: an absolute path of a file or directory on the device - track_stale: whether to bother looking for stale files (slower) + paths: The list of paths for which to list files and directories. Returns: - a four-element tuple + a set containing all files and directories contained within |paths| on the + device. + """ + nodes = set() + paths = [p.replace(' ', r'\ ') for p in paths] + command = _FILE_LIST_SCRIPT % ' '.join(paths) + current_path = "" + # We use shell=True to evaluate the command as a script through the shell, + # otherwise RunShellCommand tries to interpret it as the name of a (non + # existent) command to run. + for line in self.RunShellCommand(command, shell=True, check_return=True): + # If the line is an absolute path it's a directory, otherwise it's a file + # within the most recent directory. + if posixpath.isabs(line): + current_path = line + '/' + else: + line = current_path + line + nodes.add(line) + + return nodes + + def _GetChangedFiles(self, host_device_tuples, delete_stale=False): + """Get files to push and delete. + + Args: + host_device_tuples: a list of (host_files_path, device_files_path) tuples + to find changed files from + delete_stale: Whether to delete stale files + + Returns: + a three-element tuple 1st element: a list of (host_files_path, device_files_path) tuples to push - 2nd element: a list of host_files_path that are up-to-date - 3rd element: a list of stale files under device_path, or [] when - track_stale == False - 4th element: a cache commit function. + 2nd element: a list of missing device directories to mkdir + 3rd element: a cache commit function """ - try: - # Length calculations below assume no trailing /. - host_path = host_path.rstrip('/') - device_path = device_path.rstrip('/') - - specific_device_paths = [device_path] - ignore_other_files = not track_stale and os.path.isdir(host_path) - if ignore_other_files: - specific_device_paths = [] + # The fully expanded list of host/device tuples of files to push. + file_tuples = [] + # All directories we're pushing files to. + device_dirs_to_push_to = set() + # All files and directories we expect to have on the device after pushing + # files. + expected_device_nodes = set() + + for h, d in host_device_tuples: + assert os.path.isabs(h) and posixpath.isabs(d) + h = os.path.realpath(h) + host_path = h.rstrip('/') + device_dir = d.rstrip('/') + + expected_device_nodes.add(device_dir) + + # Add all parent directories to the directories we expect to have so we + # don't delete empty nested directories. + parent = posixpath.dirname(device_dir) + while parent and parent != '/': + expected_device_nodes.add(parent) + parent = posixpath.dirname(parent) + + if os.path.isdir(host_path): + device_dirs_to_push_to.add(device_dir) for root, _, filenames in os.walk(host_path): - relative_dir = root[len(host_path) + 1:] - specific_device_paths.extend( - posixpath.join(device_path, relative_dir, f) for f in filenames) + # ignore hidden directories + if os.path.sep + '.' in root: + continue + relative_dir = os.path.relpath(root, host_path).rstrip('.') + device_path = posixpath.join(device_dir, relative_dir).rstrip('/') + expected_device_nodes.add(device_path) + device_dirs_to_push_to.add(device_path) + files = ( + [posixpath.join(device_dir, relative_dir, f) for f in filenames]) + expected_device_nodes.update(files) + file_tuples.extend(zip( + (os.path.join(root, f) for f in filenames), files)) + else: + device_dirs_to_push_to.add(posixpath.dirname(device_dir)) + file_tuples.append((host_path, device_dir)) - def calculate_host_checksums(): - return md5sum.CalculateHostMd5Sums([host_path]) + if file_tuples or delete_stale: + current_device_nodes = self._GetDeviceNodes(device_dirs_to_push_to) + nodes_to_delete = current_device_nodes - expected_device_nodes - def calculate_device_checksums(): - if self._enable_device_files_cache: - cache_entry = self._cache['device_path_checksums'].get(device_path) - if cache_entry and cache_entry[0] == ignore_other_files: - return dict(cache_entry[1]) + missing_dirs = device_dirs_to_push_to - current_device_nodes - sums = md5sum.CalculateDeviceMd5Sums(specific_device_paths, self) + if not file_tuples: + if delete_stale and nodes_to_delete: + self.RemovePath(nodes_to_delete, force=True, recursive=True) + return (host_device_tuples, missing_dirs, lambda: 0) - cache_entry = [ignore_other_files, sums] - self._cache['device_path_checksums'][device_path] = cache_entry - return dict(sums) + possibly_stale_device_nodes = current_device_nodes - nodes_to_delete + possibly_stale_tuples = ( + [t for t in file_tuples if t[1] in possibly_stale_device_nodes]) - host_checksums, device_checksums = reraiser_thread.RunAsync(( - calculate_host_checksums, - calculate_device_checksums)) - except EnvironmentError as e: - logger.warning('Error calculating md5: %s', e) - return ([(host_path, device_path)], [], [], lambda: 0) - - to_push = [] - up_to_date = [] - to_delete = [] - if os.path.isfile(host_path): - host_checksum = host_checksums.get(host_path) - device_checksum = device_checksums.get(device_path) - if host_checksum == device_checksum: - up_to_date.append(host_path) + def calculate_host_checksums(): + # Need to compute all checksums when caching. + if self._enable_device_files_cache: + return md5sum.CalculateHostMd5Sums([t[0] for t in file_tuples]) else: - to_push.append((host_path, device_path)) - else: - for host_abs_path, host_checksum in host_checksums.iteritems(): - device_abs_path = posixpath.join( - device_path, os.path.relpath(host_abs_path, host_path)) - device_checksum = device_checksums.pop(device_abs_path, None) - if device_checksum == host_checksum: - up_to_date.append(host_abs_path) - else: - to_push.append((host_abs_path, device_abs_path)) - to_delete = device_checksums.keys() - # We can't rely solely on the checksum approach since it does not catch - # stale directories, which can result in empty directories that cause issues - # during copying in efficient_android_directory_copy.sh. So, find any stale - # directories here so they can be removed in addition to stale files. - if track_stale: - to_delete.extend(self._GetStaleDirectories(host_path, device_path)) + return md5sum.CalculateHostMd5Sums( + [t[0] for t in possibly_stale_tuples]) - def cache_commit_func(): - # When host_path is a not a directory, the path.join() call below would - # have an '' as the second argument, causing an unwanted / to be appended. - if os.path.isfile(host_path): - assert len(host_checksums) == 1 - new_sums = {device_path: host_checksums[host_path]} + def calculate_device_checksums(): + paths = set([t[1] for t in possibly_stale_tuples]) + if not paths: + return dict() + sums = dict() + if self._enable_device_files_cache: + paths_not_in_cache = set() + for path in paths: + cache_entry = self._cache['device_path_checksums'].get(path) + if cache_entry: + sums[path] = cache_entry + else: + paths_not_in_cache.add(path) + paths = paths_not_in_cache + sums.update(dict(md5sum.CalculateDeviceMd5Sums(paths, self))) + if self._enable_device_files_cache: + for path, checksum in sums.iteritems(): + self._cache['device_path_checksums'][path] = checksum + return sums + try: + host_checksums, device_checksums = reraiser_thread.RunAsync( + (calculate_host_checksums, calculate_device_checksums)) + except device_errors.CommandFailedError as e: + logger.warning('Error calculating md5: %s', e) + return (host_device_tuples, set(), lambda: 0) + + up_to_date = set() + + for host_path, device_path in possibly_stale_tuples: + device_checksum = device_checksums.get(device_path, None) + host_checksum = host_checksums.get(host_path, None) + if device_checksum == host_checksum and device_checksum is not None: + up_to_date.add(device_path) else: - new_sums = {posixpath.join(device_path, path[len(host_path) + 1:]): val - for path, val in host_checksums.iteritems()} - cache_entry = [ignore_other_files, new_sums] - self._cache['device_path_checksums'][device_path] = cache_entry + nodes_to_delete.add(device_path) - return (to_push, up_to_date, to_delete, cache_commit_func) + if delete_stale and nodes_to_delete: + self.RemovePath(nodes_to_delete, force=True, recursive=True) - def _GetStaleDirectories(self, host_path, device_path): - """Gets a list of stale directories on the device. + to_push = ( + [t for t in file_tuples if t[1] not in up_to_date]) - Args: - host_path: an absolute path of a directory on the host - device_path: an absolute path of a directory on the device + def cache_commit_func(): + if not self._enable_device_files_cache: + return + for host_path, device_path in file_tuples: + host_checksum = host_checksums.get(host_path, None) + self._cache['device_path_checksums'][device_path] = host_checksum - Returns: - A list containing absolute paths to directories on the device that are - considered stale. - """ - def get_device_dirs(path): - directories = set() - command = _RECURSIVE_DIRECTORY_LIST_SCRIPT % cmd_helper.SingleQuote(path) - # We use shell=True to evaluate the command as a script through the shell, - # otherwise RunShellCommand tries to interpret it as the name of a (non - # existent) command to run. - for line in self.RunShellCommand( - command, shell=True, check_return=True): - directories.add(posixpath.relpath(posixpath.normpath(line), path)) - return directories - - def get_host_dirs(path): - directories = set() - if not os.path.isdir(path): - return directories - for root, _, _ in os.walk(path): - if root != path: - # Strip off the top level directory so we can compare the device and - # host. - directories.add( - os.path.relpath(root, path).replace(os.sep, posixpath.sep)) - return directories - - host_dirs = get_host_dirs(host_path) - device_dirs = get_device_dirs(device_path) - stale_dirs = device_dirs - host_dirs - return [posixpath.join(device_path, d) for d in stale_dirs] + return (to_push, missing_dirs, cache_commit_func) def _ComputeDeviceChecksumsForApks(self, package_name): ret = self._cache['package_apk_checksums'].get(package_name) @@ -1749,10 +2058,11 @@ class DeviceUtils(object): def calculate_device_checksums(): return self._ComputeDeviceChecksumsForApks(package_name) - host_checksums, device_checksums = reraiser_thread.RunAsync(( - calculate_host_checksums, calculate_device_checksums)) - stale_apks = [k for (k, v) in host_checksums.iteritems() - if v not in device_checksums] + host_checksums, device_checksums = reraiser_thread.RunAsync( + (calculate_host_checksums, calculate_device_checksums)) + stale_apks = [ + k for (k, v) in host_checksums.iteritems() if v not in device_checksums + ] return stale_apks, set(host_checksums.values()) def _PushFilesImpl(self, host_device_tuples, files): @@ -1761,8 +2071,8 @@ class DeviceUtils(object): size = sum(host_utils.GetRecursiveDiskUsage(h) for h, _ in files) file_count = len(files) - dir_size = sum(host_utils.GetRecursiveDiskUsage(h) - for h, _ in host_device_tuples) + dir_size = sum( + host_utils.GetRecursiveDiskUsage(h) for h, _ in host_device_tuples) dir_file_count = 0 for h, _ in host_device_tuples: if os.path.isdir(h): @@ -1770,8 +2080,8 @@ class DeviceUtils(object): else: dir_file_count += 1 - push_duration = self._ApproximateDuration( - file_count, file_count, size, False) + push_duration = self._ApproximateDuration(file_count, file_count, size, + False) dir_push_duration = self._ApproximateDuration( len(host_device_tuples), dir_file_count, dir_size, False) zip_duration = self._ApproximateDuration(1, 1, size, True) @@ -1786,8 +2096,8 @@ class DeviceUtils(object): elif self._commands_installed is False: # Already tried and failed to install unzip command. self._PushChangedFilesIndividually(files) - elif not self._PushChangedFilesZipped( - files, [d for _, d in host_device_tuples]): + elif not self._PushChangedFilesZipped(files, + [d for _, d in host_device_tuples]): self._PushChangedFilesIndividually(files) def _MaybeInstallCommands(self): @@ -1859,13 +2169,15 @@ class DeviceUtils(object): quoted_dirs = ' '.join(cmd_helper.SingleQuote(d) for d in dirs) self.RunShellCommand( 'unzip %s&&chmod -R 777 %s' % (device_temp.name, quoted_dirs), - shell=True, as_root=True, + shell=True, + as_root=True, env={'PATH': '%s:$PATH' % install_commands.BIN_DIR}, check_return=True) return True - # TODO(nednguyen): remove this and migrate the callsite to PathExists(). + # TODO(crbug.com/1111556): remove this and migrate the callsite to + # PathExists(). @decorators.WithTimeoutAndRetriesFromInstance() def FileExists(self, device_path, timeout=None, retries=None): """Checks whether the given file exists on the device. @@ -1895,22 +2207,32 @@ class DeviceUtils(object): """ paths = device_paths if isinstance(paths, basestring): - paths = (paths,) + paths = (paths, ) if not paths: return True cmd = ['test', '-e', paths[0]] for p in paths[1:]: cmd.extend(['-a', '-e', p]) try: - self.RunShellCommand(cmd, as_root=as_root, check_return=True, - timeout=timeout, retries=retries) + self.RunShellCommand( + cmd, + as_root=as_root, + check_return=True, + timeout=timeout, + retries=retries) return True except device_errors.CommandFailedError: return False @decorators.WithTimeoutAndRetriesFromInstance() - def RemovePath(self, device_path, force=False, recursive=False, - as_root=False, rename=False, timeout=None, retries=None): + def RemovePath(self, + device_path, + force=False, + recursive=False, + as_root=False, + rename=False, + timeout=None, + retries=None): """Removes the given path(s) from the device. Args: @@ -1925,16 +2247,19 @@ class DeviceUtils(object): timeout: timeout in seconds retries: number of retries """ + def _RenamePath(path): - random_suffix = hex(random.randint(2 ** 12, 2 ** 16 - 1))[2:] + random_suffix = hex(random.randint(2**12, 2**16 - 1))[2:] dest = '%s-%s' % (path, random_suffix) try: - self.RunShellCommand( - ['mv', path, dest], as_root=as_root, check_return=True) + self.RunShellCommand(['mv', path, dest], + as_root=as_root, + check_return=True) return dest except device_errors.AdbShellCommandFailedError: # If it couldn't be moved, just try rm'ing the original path instead. return path + args = ['rm'] if force: args.append('-f') @@ -1968,7 +2293,11 @@ class DeviceUtils(object): yield device_temp @decorators.WithTimeoutAndRetriesFromInstance() - def PullFile(self, device_path, host_path, as_root=False, timeout=None, + def PullFile(self, + device_path, + host_path, + as_root=False, + timeout=None, retries=None): """Pull a file from the device. @@ -2010,8 +2339,12 @@ class DeviceUtils(object): shutil.rmtree(d) @decorators.WithTimeoutAndRetriesFromInstance() - def ReadFile(self, device_path, as_root=False, force_pull=False, - timeout=None, retries=None): + def ReadFile(self, + device_path, + as_root=False, + force_pull=False, + timeout=None, + retries=None): """Reads the contents of a file from the device. Args: @@ -2035,16 +2368,22 @@ class DeviceUtils(object): CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ + def get_size(path): return self.FileSize(path, as_root=as_root) - if (not force_pull - and 0 < get_size(device_path) <= self._MAX_ADB_OUTPUT_LENGTH): - return _JoinLines(self.RunShellCommand( - ['cat', device_path], as_root=as_root, check_return=True)) - elif as_root and self.NeedsSU(): - with self._CopyToReadableLocation(device_path) as readable_temp_file: - return self._ReadFileWithPull(readable_temp_file.name) + # Reading by pulling is faster than first getting the file size and cat-ing, + # so only read by cat when we need root. + if as_root and self.NeedsSU(): + if (not force_pull + and 0 < get_size(device_path) <= self._MAX_ADB_OUTPUT_LENGTH): + return _JoinLines( + self.RunShellCommand(['cat', device_path], + as_root=as_root, + check_return=True)) + else: + with self._CopyToReadableLocation(device_path) as readable_temp_file: + return self._ReadFileWithPull(readable_temp_file.name) else: return self._ReadFileWithPull(device_path) @@ -2055,8 +2394,13 @@ class DeviceUtils(object): self.adb.Push(host_temp.name, device_path) @decorators.WithTimeoutAndRetriesFromInstance() - def WriteFile(self, device_path, contents, as_root=False, force_push=False, - timeout=None, retries=None): + def WriteFile(self, + device_path, + contents, + as_root=False, + force_push=False, + timeout=None, + retries=None): """Writes |contents| to a file on the device. Args: @@ -2091,7 +2435,8 @@ class DeviceUtils(object): # destination files might be on different file systems (e.g. # on internal storage and an external sd card). self.RunShellCommand(['cp', device_temp.name, device_path], - as_root=True, check_return=True) + as_root=True, + check_return=True) else: # If root is not needed, we can push directly to the desired location. self._WriteFileWithPush(device_path, contents) @@ -2099,11 +2444,13 @@ class DeviceUtils(object): def _ParseLongLsOutput(self, device_path, as_root=False, **kwargs): """Run and scrape the output of 'ls -a -l' on a device directory.""" device_path = posixpath.join(device_path, '') # Force trailing '/'. - output = self.RunShellCommand( - ['ls', '-a', '-l', device_path], as_root=as_root, - check_return=True, env={'TZ': 'utc'}, **kwargs) + output = self.RunShellCommand(['ls', '-a', '-l', device_path], + as_root=as_root, + check_return=True, + env={'TZ': 'utc'}, + **kwargs) if output and output[0].startswith('total '): - output.pop(0) # pylint: disable=maybe-no-member + output.pop(0) # pylint: disable=maybe-no-member entries = [] for line in output: @@ -2278,6 +2625,7 @@ class DeviceUtils(object): Raises: CommandTimeoutError on timeout. """ + def find_property(lines, property_name): for index, line in enumerate(lines): if line.strip() == '': @@ -2360,18 +2708,21 @@ class DeviceUtils(object): def screen_density(self): """Returns the screen density of the device.""" DPI_TO_DENSITY = { - 120: 'ldpi', - 160: 'mdpi', - 240: 'hdpi', - 320: 'xhdpi', - 480: 'xxhdpi', - 640: 'xxxhdpi', + 120: 'ldpi', + 160: 'mdpi', + 240: 'hdpi', + 320: 'xhdpi', + 480: 'xxhdpi', + 640: 'xxxhdpi', } return DPI_TO_DENSITY.get(self.pixel_density, 'tvdpi') @property def pixel_density(self): - return int(self.GetProp('ro.sf.lcd_density', cache=True)) + density = self.GetProp('ro.sf.lcd_density', cache=True) + if not density and self.adb.is_emulator: + density = self.GetProp('qemu.sf.lcd_density', cache=True) + return int(density) @property def build_description(self): @@ -2402,6 +2753,15 @@ class DeviceUtils(object): return self.GetProp('ro.build.product', cache=True) @property + def build_system_root_image(self): + """Returns the system_root_image property. + + This seems to indicate whether the device is using a system-as-root + partition layout. See http://bit.ly/37F34sx for more info. + """ + return self.GetProp('ro.build.system_root_image', cache=True) + + @property def build_type(self): """Returns the build type of the system (e.g. 'user').""" return self.GetProp('ro.build.type', cache=True) @@ -2426,6 +2786,11 @@ class DeviceUtils(object): 'Invalid build version sdk: %r' % value) @property + def tracing_path(self): + """Returns the tracing path of the device for atrace.""" + return self.GetTracingPath() + + @property def product_cpu_abi(self): """Returns the product cpu abi of the device (e.g. 'armeabi-v7a'). @@ -2435,6 +2800,11 @@ class DeviceUtils(object): return self.GetProp('ro.product.cpu.abi', cache=True) @property + def product_cpu_abis(self): + """Returns all product cpu abi of the device.""" + return self.GetProp('ro.product.cpu.abilist', cache=True).split(',') + + @property def product_model(self): """Returns the name of the product model (e.g. 'Nexus 7').""" return self.GetProp('ro.product.model', cache=True) @@ -2459,13 +2829,10 @@ class DeviceUtils(object): # Change the token every time to ensure that it will match only the # previously dumped cache. token = str(uuid.uuid1()) - cmd = ( - 'c=/data/local/tmp/cache_token;' - 'echo $EXTERNAL_STORAGE;' - 'cat $c 2>/dev/null||echo;' - 'echo "%s">$c &&' % token + - 'getprop' - ) + cmd = ('c=/data/local/tmp/cache_token;' + 'echo $EXTERNAL_STORAGE;' + 'cat $c 2>/dev/null||echo;' + 'echo "%s">$c &&' % token + 'getprop') output = self.RunShellCommand( cmd, shell=True, check_return=True, large_output=True) # Error-checking for this existing is done in GetExternalStoragePath(). @@ -2480,6 +2847,39 @@ class DeviceUtils(object): self._cache['token'] = token @decorators.WithTimeoutAndRetriesFromInstance() + def GetTracingPath(self, timeout=None, retries=None): + """Gets tracing path from the device. + + Args: + timeout: timeout in seconds + retries: number of retries + + Returns: + /sys/kernel/debug/tracing for device with debugfs mount support; + /sys/kernel/tracing for device with tracefs support; + /sys/kernel/debug/tracing if support can't be determined. + + Raises: + CommandTimeoutError on timeout. + """ + tracing_path = self._cache['tracing_path'] + if tracing_path: + return tracing_path + with self._cache_lock: + tracing_path = '/sys/kernel/debug/tracing' + try: + lines = self.RunShellCommand(['mount'], + check_return=True, + timeout=timeout, + retries=retries) + if not any('debugfs' in line for line in lines): + tracing_path = '/sys/kernel/tracing' + except device_errors.AdbCommandFailedError: + pass + self._cache['tracing_path'] = tracing_path + return tracing_path + + @decorators.WithTimeoutAndRetriesFromInstance() def GetProp(self, property_name, cache=False, timeout=None, retries=None): """Gets a property from the device. @@ -2496,8 +2896,9 @@ class DeviceUtils(object): Raises: CommandTimeoutError on timeout. """ - assert isinstance(property_name, basestring), ( - "property_name is not a string: %r" % property_name) + assert isinstance( + property_name, + basestring), ("property_name is not a string: %r" % property_name) if cache: # It takes ~120ms to query a single property, and ~130ms to query all @@ -2506,15 +2907,21 @@ class DeviceUtils(object): else: # timeout and retries are handled down at run shell, because we don't # want to apply them in the other branch when reading from the cache - value = self.RunShellCommand( - ['getprop', property_name], single_line=True, check_return=True, - timeout=timeout, retries=retries) + value = self.RunShellCommand(['getprop', property_name], + single_line=True, + check_return=True, + timeout=timeout, + retries=retries) self._cache['getprop'][property_name] = value # Non-existent properties are treated as empty strings by getprop. return self._cache['getprop'].get(property_name, '') @decorators.WithTimeoutAndRetriesFromInstance() - def SetProp(self, property_name, value, check=False, timeout=None, + def SetProp(self, + property_name, + value, + check=False, + timeout=None, retries=None): """Sets a property on the device. @@ -2533,20 +2940,21 @@ class DeviceUtils(object): set on the device (e.g. because it is not rooted). CommandTimeoutError on timeout. """ - assert isinstance(property_name, basestring), ( - "property_name is not a string: %r" % property_name) + assert isinstance( + property_name, + basestring), ("property_name is not a string: %r" % property_name) assert isinstance(value, basestring), "value is not a string: %r" % value self.RunShellCommand(['setprop', property_name, value], check_return=True) prop_cache = self._cache['getprop'] if property_name in prop_cache: del prop_cache[property_name] - # TODO(perezju) remove the option and make the check mandatory, but using a - # single shell script to both set- and getprop. + # TODO(crbug.com/1029772) remove the option and make the check mandatory, + # but using a single shell script to both set- and getprop. if check and value != self.GetProp(property_name, cache=False): raise device_errors.CommandFailedError( - 'Unable to set property %r on the device to %r' - % (property_name, value), str(self)) + 'Unable to set property %r on the device to %r' % (property_name, + value), str(self)) @decorators.WithTimeoutAndRetriesFromInstance() def GetABI(self, timeout=None, retries=None): @@ -2565,6 +2973,12 @@ class DeviceUtils(object): """ return self.GetProp('ro.product.cpu.abi', cache=True) + @decorators.WithTimeoutAndRetriesFromInstance() + def GetFeatures(self, timeout=None, retries=None): + """Returns the features supported on the device.""" + lines = self.RunShellCommand(['pm', 'list', 'features'], check_return=True) + return [f[8:] for f in lines if f.startswith('feature:')] + def _GetPsOutput(self, pattern): """Runs |ps| command on the device and returns its output, @@ -2677,8 +3091,11 @@ class DeviceUtils(object): return procs_pids @decorators.WithTimeoutAndRetriesFromInstance() - def GetApplicationPids(self, process_name, at_most_one=False, - timeout=None, retries=None): + def GetApplicationPids(self, + process_name, + at_most_one=False, + timeout=None, + retries=None): """Returns the PID or PIDs of a given process name. Note that the |process_name|, often the package name, must match exactly. @@ -2700,13 +3117,15 @@ class DeviceUtils(object): CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ - pids = [p.pid for p in self.ListProcesses(process_name) - if p.name == process_name] + pids = [ + p.pid for p in self.ListProcesses(process_name) + if p.name == process_name + ] if at_most_one: if len(pids) > 1: raise device_errors.CommandFailedError( - 'Expected a single PID for %r but found: %r.' % ( - process_name, pids), + 'Expected a single PID for %r but found: %r.' % (process_name, + pids), device_serial=str(self)) return pids[0] if pids else None else: @@ -2728,8 +3147,9 @@ class DeviceUtils(object): CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ - output = self.RunShellCommand( - ['getenforce'], check_return=True, single_line=True).lower() + output = self.RunShellCommand(['getenforce'], + check_return=True, + single_line=True).lower() if output not in _SELINUX_MODE: raise device_errors.CommandFailedError( 'Unexpected getenforce output: %s' % output) @@ -2750,9 +3170,9 @@ class DeviceUtils(object): CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ - self.RunShellCommand( - ['setenforce', '1' if int(enabled) else '0'], as_root=True, - check_return=True) + self.RunShellCommand(['setenforce', '1' if int(enabled) else '0'], + as_root=True, + check_return=True) @decorators.WithTimeoutAndRetriesFromInstance() def GetWebViewUpdateServiceDump(self, timeout=None, retries=None): @@ -2780,8 +3200,8 @@ class DeviceUtils(object): if self.build_version_sdk < version_codes.OREO: return result - output = self.RunShellCommand( - ['dumpsys', 'webviewupdate'], check_return=True) + output = self.RunShellCommand(['dumpsys', 'webviewupdate'], + check_return=True) webview_packages = {} for line in output: match = re.search(_WEBVIEW_SYSUPDATE_CURRENT_PKG_RE, line) @@ -2832,8 +3252,7 @@ class DeviceUtils(object): CommandTimeoutError on timeout. DeviceUnreachableError on missing device. """ - installed = self.GetApplicationPaths(package_name) - if not installed: + if not self.IsApplicationInstalled(package_name): raise device_errors.CommandFailedError( '%s is not installed' % package_name, str(self)) output = self.RunShellCommand( @@ -2861,9 +3280,20 @@ class DeviceUtils(object): '%s does not declare a WebView native library, so it cannot ' 'be a WebView provider' % package_name, str(self)) if re.search(r'SDK version too low', reason): - raise device_errors.CommandFailedError( - '%s needs a higher targetSdkVersion (must be >= %d)' % - (package_name, self.build_version_sdk), str(self)) + app_target_sdk_version = self.GetApplicationTargetSdk(package_name) + is_preview_sdk = self.GetProp('ro.build.version.preview_sdk') == '1' + if is_preview_sdk: + codename = self.GetProp('ro.build.version.codename') + raise device_errors.CommandFailedError( + '%s targets a finalized SDK (%r), but valid WebView providers ' + 'must target a pre-finalized SDK (%r) on this device' % + (package_name, app_target_sdk_version, codename), str(self)) + else: + raise device_errors.CommandFailedError( + '%s has targetSdkVersion %r, but valid WebView providers must ' + 'target >= %r on this device' % + (package_name, app_target_sdk_version, self.build_version_sdk), + str(self)) if re.search(r'Version code too low', reason): raise device_errors.CommandFailedError( '%s needs a higher versionCode (must be >= %d)' % @@ -2920,8 +3350,10 @@ class DeviceUtils(object): # redundant-packages is the opposite of fallback logic enable_string = 'disable' if enabled else 'enable' output = self.RunShellCommand( - ['cmd', 'webviewupdate', '%s-redundant-packages' % enable_string], - single_line=True, check_return=True) + ['cmd', 'webviewupdate', + '%s-redundant-packages' % enable_string], + single_line=True, + check_return=True) if output == 'Success': logging.info('WebView Fallback Logic is %s', 'enabled' if enabled else 'disabled') @@ -2949,8 +3381,8 @@ class DeviceUtils(object): DeviceUnreachableError on missing device. """ if not host_path: - host_path = os.path.abspath('screenshot-%s-%s.png' % ( - self.serial, _GetTimeStamp())) + host_path = os.path.abspath( + 'screenshot-%s-%s.png' % (self.serial, _GetTimeStamp())) with device_temp_file.DeviceTempFile(self.adb, suffix='.png') as device_tmp: self.RunShellCommand(['/system/bin/screencap', '-p', device_tmp.name], check_return=True) @@ -2964,13 +3396,15 @@ class DeviceUtils(object): Returns: Name of the crashed package if a dialog is focused, None otherwise. """ + def _FindFocusedWindow(): match = None # TODO(jbudorick): Try to grep the output on the device instead of using # large_output if/when DeviceUtils exposes a public interface for piped # shell command handling. for line in self.RunShellCommand(['dumpsys', 'window', 'windows'], - check_return=True, large_output=True): + check_return=True, + large_output=True): match = re.match(_CURRENT_FOCUS_CRASH_RE, line) if match: break @@ -3017,13 +3451,15 @@ class DeviceUtils(object): 'package_apk_checksums': {}, # Map of property_name -> value 'getprop': {}, - # Map of device_path -> [ignore_other_files, map of path->checksum] + # Map of device path -> checksum] 'device_path_checksums': {}, # Location of sdcard ($EXTERNAL_STORAGE). 'external_storage': None, # Token used to detect when LoadCacheData is stale. 'token': None, 'prev_token': None, + # Path for tracing. + 'tracing_path': None, } @decorators.WithTimeoutAndRetriesFromInstance() @@ -3042,7 +3478,11 @@ class DeviceUtils(object): Returns: Whether the cache was loaded. """ - obj = json.loads(data) + try: + obj = json.loads(data) + except ValueError: + logger.error('Unable to parse cache file. Not using it.') + return False self._EnsureCacheInitialized() given_token = obj.get('token') if not given_token or self._cache['prev_token'] != given_token: @@ -3106,17 +3546,25 @@ class DeviceUtils(object): return parallelizer.SyncParallelizer(devices) @classmethod - def HealthyDevices(cls, blacklist=None, device_arg='default', retries=1, - enable_usb_resets=False, abis=None, **kwargs): + def HealthyDevices(cls, + denylist=None, + device_arg='default', + retries=1, + enable_usb_resets=False, + abis=None, + # TODO(crbug.com/1097306): Remove this once clients have + # stopped passing it. + blacklist=None, + **kwargs): """Returns a list of DeviceUtils instances. - Returns a list of DeviceUtils instances that are attached, not blacklisted, + Returns a list of DeviceUtils instances that are attached, not denylisted, and optionally filtered by --device flags or ANDROID_SERIAL environment variable. Args: - blacklist: A DeviceBlacklist instance (optional). Device serials in this - blacklist will never be returned, but a warning will be logged if they + denylist: A DeviceDenylist instance (optional). Device serials in this + denylist will never be returned, but a warning will be logged if they otherwise would have been. device_arg: The value of the --device flag. This can be: 'default' -> Same as [], but returns an empty list rather than raise a @@ -3126,9 +3574,9 @@ class DeviceUtils(object): attached device. Raises an exception if multiple devices are attached. 'serial' -> Returns an instance for the given serial, if not - blacklisted. + denylisted. ['A', 'B', ...] -> Returns instances for the subset that is not - blacklisted. + denylisted. retries: Number of times to restart adb server and query it again if no devices are found on the previous attempts, with exponential backoffs up to 60s between each retry. @@ -3143,7 +3591,7 @@ class DeviceUtils(object): A list of DeviceUtils instances. Raises: - NoDevicesError: Raised when no non-blacklisted devices exist and + NoDevicesError: Raised when no non-denylisted devices exist and device_arg is passed. MultipleDevicesError: Raise when multiple devices exist, but |device_arg| is None. @@ -3157,18 +3605,22 @@ class DeviceUtils(object): if not (isinstance(device_arg, tuple) or isinstance(device_arg, list)): select_multiple = False if device_arg: - device_arg = (device_arg,) + device_arg = (device_arg, ) + + # TODO(crbug.com/1097306): Remove this once clients have switched. + if blacklist and not denylist: + denylist = blacklist - blacklisted_devices = blacklist.Read() if blacklist else [] + denylisted_devices = denylist.Read() if denylist else [] # adb looks for ANDROID_SERIAL, so support it as well. android_serial = os.environ.get('ANDROID_SERIAL') if not device_arg and android_serial: - device_arg = (android_serial,) + device_arg = (android_serial, ) - def blacklisted(serial): - if serial in blacklisted_devices: - logger.warning('Device %s is blacklisted.', serial) + def denylisted(serial): + if serial in denylisted_devices: + logger.warning('Device %s is denylisted.', serial) return True return False @@ -3180,12 +3632,12 @@ class DeviceUtils(object): def _get_devices(): if device_arg: - devices = [cls(x, **kwargs) for x in device_arg if not blacklisted(x)] + devices = [cls(x, **kwargs) for x in device_arg if not denylisted(x)] else: devices = [] for adb in adb_wrapper.AdbWrapper.Devices(): serial = adb.GetDeviceSerial() - if not blacklisted(serial): + if not denylisted(serial): device = cls(_CreateAdbWrapper(adb), **kwargs) if supports_abi(device.GetABI(), serial): devices.append(device) @@ -3208,7 +3660,7 @@ class DeviceUtils(object): else: reset_usb.reset_all_android_devices() - for attempt in xrange(retries+1): + for attempt in xrange(retries + 1): try: return _get_devices() except device_errors.NoDevicesError: @@ -3233,30 +3685,26 @@ class DeviceUtils(object): logger.info('Restarting adbd on device.') with device_temp_file.DeviceTempFile(self.adb, suffix='.sh') as script: self.WriteFile(script.name, _RESTART_ADBD_SCRIPT) - self.RunShellCommand( - ['source', script.name], check_return=True, as_root=True) + self.RunShellCommand(['source', script.name], + check_return=True, + as_root=True) self.adb.WaitForDevice() @decorators.WithTimeoutAndRetriesFromInstance() def GrantPermissions(self, package, permissions, timeout=None, retries=None): - # Permissions only need to be set on M and above because of the changes to - # the permission model. - if not permissions or self.build_version_sdk < version_codes.MARSHMALLOW: + if not permissions: return - permissions = set( - p for p in permissions if not _PERMISSIONS_BLACKLIST_RE.match(p)) + permissions = set(p for p in permissions + if not _PERMISSIONS_DENYLIST_RE.match(p)) if ('android.permission.WRITE_EXTERNAL_STORAGE' in permissions and 'android.permission.READ_EXTERNAL_STORAGE' not in permissions): permissions.add('android.permission.READ_EXTERNAL_STORAGE') script = ';'.join([ - 'p={package}', - 'for q in {permissions}', - 'do pm grant "$p" "$q"', - 'echo "{sep}$q{sep}$?{sep}"', - 'done' + 'p={package}', 'for q in {permissions}', 'do pm grant "$p" "$q"', + 'echo "{sep}$q{sep}$?{sep}"', 'done' ]).format( package=cmd_helper.SingleQuote(package), permissions=' '.join( @@ -3265,17 +3713,21 @@ class DeviceUtils(object): logger.info('Setting permissions for %s.', package) res = self.RunShellCommand( - script, shell=True, raw_output=True, large_output=True, + script, + shell=True, + raw_output=True, + large_output=True, check_return=True) res = res.split(_SHELL_OUTPUT_SEPARATOR) failures = [ - (permission, output.strip()) - for permission, status, output in zip(res[1::3], res[2::3], res[0::3]) - if int(status)] + (permission, output.strip()) + for permission, status, output in zip(res[1::3], res[2::3], res[0::3]) + if int(status) + ] if failures: logger.warning( - 'Failed to grant some permissions. Blacklist may need to be updated?') + 'Failed to grant some permissions. Denylist may need to be updated?') for permission, output in failures: # Try to grab the relevant error message from the output. m = _PERMISSIONS_EXCEPTION_RE.search(output) @@ -3316,8 +3768,8 @@ class DeviceUtils(object): dumpsys_out = self._RunPipedShellCommand( 'dumpsys input_method | grep %s' % input_check) if not dumpsys_out: - raise device_errors.CommandFailedError( - 'Unable to detect screen state', str(self)) + raise device_errors.CommandFailedError('Unable to detect screen state', + str(self)) return check_value in dumpsys_out[0] @decorators.WithTimeoutAndRetriesFromInstance() @@ -3327,6 +3779,7 @@ class DeviceUtils(object): Args: on: bool to decide state to switch to. True = on False = off. """ + def screen_test(): return self.IsScreenOn() == on @@ -3353,7 +3806,10 @@ class DeviceUtils(object): self.RunShellCommand(['chown', owner_group] + paths, check_return=True) @decorators.WithTimeoutAndRetriesFromInstance() - def ChangeSecurityContext(self, security_context, paths, timeout=None, + def ChangeSecurityContext(self, + security_context, + paths, + timeout=None, retries=None): """Changes the SELinux security context for files. diff --git a/catapult/devil/devil/android/device_utils_devicetest.py b/catapult/devil/devil/android/device_utils_devicetest.py index 0836f3ea..a0734c2a 100755 --- a/catapult/devil/devil/android/device_utils_devicetest.py +++ b/catapult/devil/devil/android/device_utils_devicetest.py @@ -2,7 +2,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Unit tests for the contents of device_utils.py (mostly DeviceUtils). The test will invoke real devices @@ -16,7 +15,11 @@ import unittest if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', ))) + os.path.abspath(os.path.join( + os.path.dirname(__file__), + '..', + '..', + ))) from devil.android import device_test_case from devil.android import device_utils @@ -32,7 +35,6 @@ _SUB_DIR2 = "sub2" class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): - def setUp(self): super(DeviceUtilsPushDeleteFilesTest, self).setUp() self.adb = adb_wrapper.AdbWrapper(self.serial) @@ -88,8 +90,9 @@ class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): device_file_path = "%s/%s" % (_DEVICE_DIR, file_name) self.adb.Push(host_file_path, device_file_path) self.device.PushChangedFiles([(host_file_path, device_file_path)]) - result = self.device.RunShellCommand( - ['cat', device_file_path], check_return=True, single_line=True) + result = self.device.RunShellCommand(['cat', device_file_path], + check_return=True, + single_line=True) self.assertEqual(_OLD_CONTENTS, result) cmd_helper.RunCmd(['rm', host_file_path]) @@ -103,8 +106,9 @@ class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): with open(host_file_path, 'w') as f: f.write(_NEW_CONTENTS) self.device.PushChangedFiles([(host_file_path, device_file_path)]) - result = self.device.RunShellCommand( - ['cat', device_file_path], check_return=True, single_line=True) + result = self.device.RunShellCommand(['cat', device_file_path], + check_return=True, + single_line=True) self.assertEqual(_NEW_CONTENTS, result) cmd_helper.RunCmd(['rm', host_file_path]) @@ -144,9 +148,10 @@ class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): cmd_helper.RunCmd(['rm', host_file_path2]) self.device.PushChangedFiles([(host_tmp_dir, _DEVICE_DIR)], - delete_device_stale=True) - result = self.device.RunShellCommand( - ['cat', device_file_path1], check_return=True, single_line=True) + delete_device_stale=True) + result = self.device.RunShellCommand(['cat', device_file_path1], + check_return=True, + single_line=True) self.assertEqual(_NEW_CONTENTS, result) filenames = self.device.ListDirectory(_DEVICE_DIR) @@ -174,8 +179,8 @@ class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): device_file_path1 = "%s/%s" % (_DEVICE_DIR, file_name1) device_file_path2 = "%s/%s" % (_DEVICE_DIR, file_name2) device_file_path3 = "%s/%s/%s" % (_DEVICE_DIR, _SUB_DIR1, file_name3) - device_file_path4 = "%s/%s/%s/%s" % (_DEVICE_DIR, _SUB_DIR, - _SUB_DIR2, file_name4) + device_file_path4 = "%s/%s/%s/%s" % (_DEVICE_DIR, _SUB_DIR, _SUB_DIR2, + file_name4) self.adb.Push(host_file_path1, device_file_path1) self.adb.Push(host_file_path2, device_file_path2) @@ -188,9 +193,10 @@ class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): cmd_helper.RunCmd(['rm', host_file_path4]) self.device.PushChangedFiles([(host_tmp_dir, _DEVICE_DIR)], - delete_device_stale=True) - result = self.device.RunShellCommand( - ['cat', device_file_path1], check_return=True, single_line=True) + delete_device_stale=True) + result = self.device.RunShellCommand(['cat', device_file_path1], + check_return=True, + single_line=True) self.assertEqual(_NEW_CONTENTS, result) filenames = self.device.ListDirectory(_DEVICE_DIR) @@ -199,8 +205,9 @@ class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): self.assertIn(_SUB_DIR, filenames) self.assertEqual(3, len(filenames)) - result = self.device.RunShellCommand( - ['cat', device_file_path3], check_return=True, single_line=True) + result = self.device.RunShellCommand(['cat', device_file_path3], + check_return=True, + single_line=True) self.assertEqual(_OLD_CONTENTS, result) filenames = self.device.ListDirectory( @@ -223,7 +230,7 @@ class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): # Push all our created files/directories and verify they're on the device. self.device.PushChangedFiles([(host_tmp_dir, _DEVICE_DIR)], - delete_device_stale=True) + delete_device_stale=True) top_level_dirs = self.device.ListDirectory(_DEVICE_DIR) self.assertIn(_SUB_DIR1, top_level_dirs) self.assertIn(_SUB_DIR, top_level_dirs) @@ -233,7 +240,7 @@ class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): # Remove one of the directories on the host and push again. cmd_helper.RunCmd(['rm', '-rf', host_sub_dir2]) self.device.PushChangedFiles([(host_tmp_dir, _DEVICE_DIR)], - delete_device_stale=True) + delete_device_stale=True) # Verify that the directory we removed is no longer on the device, but the # other directories still are. @@ -266,7 +273,6 @@ class DeviceUtilsPushDeleteFilesTest(device_test_case.DeviceTestCase): class PsOutputCompatibilityTests(device_test_case.DeviceTestCase): - def setUp(self): super(PsOutputCompatibilityTests, self).setUp() self.adb = adb_wrapper.AdbWrapper(self.serial) @@ -283,8 +289,8 @@ class PsOutputCompatibilityTests(device_test_case.DeviceTestCase): column = column.upper() self.assertEqual( header[idx], column, - 'Expected column %s at index %d but found %s\nsource: %r' % ( - column, idx, header[idx], lines[0])) + 'Expected column %s at index %d but found %s\nsource: %r' % + (column, idx, header[idx], lines[0])) # Check pid and ppid are numeric values. for line in lines[1:]: @@ -293,8 +299,8 @@ class PsOutputCompatibilityTests(device_test_case.DeviceTestCase): for key in ('pid', 'ppid'): self.assertTrue( row[key].isdigit(), - 'Expected numeric %s value but found %r\nsource: %r' % ( - key, row[key], line)) + 'Expected numeric %s value but found %r\nsource: %r' % + (key, row[key], line)) if __name__ == '__main__': diff --git a/catapult/devil/devil/android/device_utils_test.py b/catapult/devil/devil/android/device_utils_test.py index 5799c7b8..38e64a97 100755 --- a/catapult/devil/devil/android/device_utils_test.py +++ b/catapult/devil/devil/android/device_utils_test.py @@ -2,7 +2,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Unit tests for the contents of device_utils.py (mostly DeviceUtils). """ @@ -10,10 +9,12 @@ Unit tests for the contents of device_utils.py (mostly DeviceUtils). # pylint: disable=protected-access # pylint: disable=unused-argument +import collections import contextlib import json import logging import os +import posixpath import stat import sys import unittest @@ -30,9 +31,17 @@ from devil.android.sdk import version_codes from devil.utils import cmd_helper from devil.utils import mock_calls +with devil_env.SysPath( + os.path.join(devil_env.CATAPULT_ROOT_PATH, 'common', 'py_utils')): + from py_utils import tempfile_ext + with devil_env.SysPath(devil_env.PYMOCK_PATH): import mock # pylint: disable=import-error +TEST_APK_PATH = '/fake/test/app.apk' +TEST_PACKAGE = 'test.package' + + def Process(name, pid, ppid='1'): return device_utils.ProcessInfo(name=name, pid=pid, ppid=ppid) @@ -52,13 +61,24 @@ class AnyStringWith(object): return '<AnyStringWith: %s>' % self._value -class _MockApkHelper(object): +class _FakeContextManager(object): + def __init__(self, obj): + self._obj = obj + + def __enter__(self): + return self._obj + + def __exit__(self, type_, value, traceback): + pass - def __init__(self, path, package_name, perms=None): + +class _MockApkHelper(object): + def __init__(self, path, package_name, perms=None, splits=None): self.path = path self.is_bundle = path.endswith('_bundle') self.package_name = package_name self.perms = perms + self.splits = splits if splits else [] self.abis = [abis.ARM] def GetPackageName(self): @@ -70,13 +90,24 @@ class _MockApkHelper(object): def GetAbis(self): return self.abis + def GetApkPaths(self, + device, + modules=None, + allow_cached_props=False, + additional_locales=None): + return _FakeContextManager([self.path] + self.splits) + + #override + @staticmethod + def SupportsSplits(): + return True + class _MockMultipleDevicesError(Exception): pass class DeviceUtilsInitTest(unittest.TestCase): - def testInitWithStr(self): serial_as_str = str('0123456789abcdef') d = device_utils.DeviceUtils('0123456789abcdef') @@ -101,17 +132,13 @@ class DeviceUtilsInitTest(unittest.TestCase): class DeviceUtilsGetAVDsTest(mock_calls.TestCase): - def testGetAVDs(self): - mocked_attrs = { - 'android_sdk': '/my/sdk/path' - } + mocked_attrs = {'android_sdk': '/my/sdk/path'} with mock.patch('devil.devil_env._Environment.LocalPath', mock.Mock(side_effect=lambda a: mocked_attrs[a])): with self.assertCall( mock.call.devil.utils.cmd_helper.GetCmdOutput( - [mock.ANY, 'list', 'avd']), - 'Available Android Virtual Devices:\n' + [mock.ANY, 'list', 'avd']), 'Available Android Virtual Devices:\n' ' Name: my_android5.0\n' ' Path: /some/path/to/.android/avd/my_android5.0.avd\n' ' Target: Android 5.0 (API level 21)\n' @@ -121,26 +148,21 @@ class DeviceUtilsGetAVDsTest(mock_calls.TestCase): class DeviceUtilsRestartServerTest(mock_calls.TestCase): - @mock.patch('time.sleep', mock.Mock()) def testRestartServer_succeeds(self): with self.assertCalls( mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.KillServer(), (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutput( - ['pgrep', 'adb']), - (1, '')), + ['pgrep', 'adb']), (1, '')), mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.StartServer(), (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutput( ['pgrep', 'adb']), - (1, '')), - (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutput( - ['pgrep', 'adb']), - (0, '123\n'))): + (1, '')), (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutput( + ['pgrep', 'adb']), (0, '123\n'))): device_utils.RestartServer() class MockTempFile(object): - def __init__(self, name='/tmp/some/file'): self.file = mock.MagicMock(spec=file) self.file.name = name @@ -160,7 +182,6 @@ class MockTempFile(object): class MockLogger(mock.Mock): def __init__(self, *args, **kwargs): super(MockLogger, self).__init__(*args, **kwargs) - # TODO(perezju): Consider adding traps for error, info, etc. self.warnings = [] def warning(self, message, *args): @@ -173,7 +194,6 @@ def PatchLogger(): class _PatchedFunction(object): - def __init__(self, patched=None, mocked=None): self.patched = patched self.mocked = mocked @@ -188,7 +208,6 @@ def _AdbWrapperMock(test_serial, is_ready=True): class DeviceUtilsTest(mock_calls.TestCase): - def setUp(self): self.adb = _AdbWrapperMock('0123456789abcdef') self.device = device_utils.DeviceUtils( @@ -198,19 +217,21 @@ class DeviceUtilsTest(mock_calls.TestCase): def AdbCommandError(self, args=None, output=None, status=None, msg=None): if args is None: args = ['[unspecified]'] - return mock.Mock(side_effect=device_errors.AdbCommandFailedError( - args, output, status, msg, str(self.device))) + return mock.Mock( + side_effect=device_errors.AdbCommandFailedError(args, output, status, + msg, str(self.device))) def CommandError(self, msg=None): if msg is None: msg = 'Command failed' - return mock.Mock(side_effect=device_errors.CommandFailedError( - msg, str(self.device))) + return mock.Mock( + side_effect=device_errors.CommandFailedError(msg, str(self.device))) def ShellError(self, output=None, status=1): def action(cmd, *args, **kwargs): - raise device_errors.AdbShellCommandFailedError( - cmd, output, status, str(self.device)) + raise device_errors.AdbShellCommandFailedError(cmd, output, status, + str(self.device)) + if output is None: output = 'Permission denied\n' return action @@ -218,19 +239,20 @@ class DeviceUtilsTest(mock_calls.TestCase): def TimeoutError(self, msg=None): if msg is None: msg = 'Operation timed out' - return mock.Mock(side_effect=device_errors.CommandTimeoutError( - msg, str(self.device))) + return mock.Mock( + side_effect=device_errors.CommandTimeoutError(msg, str(self.device))) def EnsureCacheInitialized(self, props=None, sdcard='/sdcard'): props = props or [] ret = [sdcard, 'TOKEN'] + props return (self.call.device.RunShellCommand( AnyStringWith('getprop'), - shell=True, check_return=True, large_output=True), ret) + shell=True, + check_return=True, + large_output=True), ret) class DeviceUtilsEqTest(DeviceUtilsTest): - def testEq_equal_deviceUtils(self): other = device_utils.DeviceUtils(_AdbWrapperMock('0123456789abcdef')) self.assertTrue(self.device == other) @@ -260,7 +282,6 @@ class DeviceUtilsEqTest(DeviceUtilsTest): class DeviceUtilsLtTest(DeviceUtilsTest): - def testLt_lessThan(self): other = device_utils.DeviceUtils(_AdbWrapperMock('ffffffffffffffff')) self.assertTrue(self.device < other) @@ -289,15 +310,13 @@ class DeviceUtilsLtTest(DeviceUtilsTest): class DeviceUtilsStrTest(DeviceUtilsTest): - def testStr_returnsSerial(self): - with self.assertCalls( - (self.call.adb.GetDeviceSerial(), '0123456789abcdef')): + with self.assertCalls((self.call.adb.GetDeviceSerial(), + '0123456789abcdef')): self.assertEqual('0123456789abcdef', str(self.device)) class DeviceUtilsIsOnlineTest(DeviceUtilsTest): - def testIsOnline_true(self): with self.assertCall(self.call.adb.GetState(), 'device'): self.assertTrue(self.device.IsOnline()) @@ -312,85 +331,136 @@ class DeviceUtilsIsOnlineTest(DeviceUtilsTest): class DeviceUtilsHasRootTest(DeviceUtilsTest): - def testHasRoot_true(self): - with self.patch_call(self.call.device.build_type, - return_value='userdebug'), ( - self.patch_call(self.call.device.product_name, - return_value='notasailfish')), ( - self.assertCall(self.call.adb.Shell('ls /root'), 'foo\n')): + with self.patch_call( + self.call.device.build_type, + return_value='userdebug'), (self.patch_call( + self.call.device.build_system_root_image, + return_value='')), (self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.PIE)), (self.patch_call( + self.call.device.product_name, + return_value='notasailfish')), (self.assertCall( + self.call.adb.Shell('ls /root'), 'foo\n')): self.assertTrue(self.device.HasRoot()) def testhasRootSpecial_true(self): - with self.patch_call(self.call.device.build_type, - return_value='userdebug'), ( - self.patch_call(self.call.device.product_name, - return_value='sailfish')), ( - self.assertCall(self.call.adb.Shell('getprop service.adb.root'), + with self.patch_call( + self.call.device.build_type, + return_value='userdebug'), (self.patch_call( + self.call.device.build_system_root_image, + return_value='')), (self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.PIE)), (self.patch_call( + self.call.device.product_name, + return_value='sailfish')), (self.assertCall( + self.call.adb.Shell('getprop service.adb.root'), '1\n')): self.assertTrue(self.device.HasRoot()) def testhasRootSpecialAosp_true(self): - with self.patch_call(self.call.device.build_type, - return_value='userdebug'), ( - self.patch_call(self.call.device.product_name, - return_value='aosp_sailfish')), ( - self.assertCall(self.call.adb.Shell('getprop service.adb.root'), + with self.patch_call( + self.call.device.build_type, + return_value='userdebug'), (self.patch_call( + self.call.device.build_system_root_image, + return_value='')), (self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.PIE)), (self.patch_call( + self.call.device.product_name, + return_value='aosp_sailfish')), (self.assertCall( + self.call.adb.Shell('getprop service.adb.root'), '1\n')): self.assertTrue(self.device.HasRoot()) def testhasRootEngBuild_true(self): - with self.patch_call(self.call.device.build_type, - return_value='eng'): + with self.patch_call(self.call.device.build_type, return_value='eng'): self.assertTrue(self.device.HasRoot()) def testHasRoot_false(self): - with self.patch_call(self.call.device.build_type, - return_value='userdebug'), ( - self.patch_call(self.call.device.product_name, - return_value='notasailfish')), ( - self.assertCall(self.call.adb.Shell('ls /root'), - self.ShellError())): + with self.patch_call( + self.call.device.build_type, + return_value='userdebug'), (self.patch_call( + self.call.device.build_system_root_image, + return_value='')), (self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.PIE)), (self.patch_call( + self.call.device.product_name, + return_value='notasailfish')), (self.assertCall( + self.call.adb.Shell('ls /root'), self.ShellError())): self.assertFalse(self.device.HasRoot()) def testHasRootSpecial_false(self): - with self.patch_call(self.call.device.build_type, - return_value='userdebug'), ( - self.patch_call(self.call.device.product_name, - return_value='sailfish')), ( - self.assertCall(self.call.adb.Shell('getprop service.adb.root'), - '\n')): + with self.patch_call( + self.call.device.build_type, + return_value='userdebug'), (self.patch_call( + self.call.device.build_system_root_image, + return_value='')), (self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.PIE)), (self.patch_call( + self.call.device.product_name, + return_value='sailfish')), (self.assertCall( + self.call.adb.Shell('getprop service.adb.root'), '\n')): self.assertFalse(self.device.HasRoot()) def testHasRootSpecialAosp_false(self): - with self.patch_call(self.call.device.build_type, - return_value='userdebug'), ( - self.patch_call(self.call.device.product_name, - return_value='aosp_sailfish')), ( - self.assertCall(self.call.adb.Shell('getprop service.adb.root'), - '\n')): + with self.patch_call( + self.call.device.build_type, + return_value='userdebug'), (self.patch_call( + self.call.device.build_system_root_image, + return_value='')), (self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.PIE)), (self.patch_call( + self.call.device.product_name, + return_value='aosp_sailfish')), (self.assertCall( + self.call.adb.Shell('getprop service.adb.root'), '\n')): self.assertFalse(self.device.HasRoot()) -class DeviceUtilsEnableRootTest(DeviceUtilsTest): + def testHasRootSystemRootImage(self): + with self.patch_call( + self.call.device.build_type, + return_value='userdebug'), (self.patch_call( + self.call.device.build_system_root_image, + return_value='true')), (self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.PIE)), (self.patch_call( + self.call.device.product_name, + return_value='notasailfish')), (self.assertCall( + self.call.adb.Shell('getprop service.adb.root'), + '1\n')): + self.assertTrue(self.device.HasRoot()) + def testHasRoot10(self): + # All devices on Android 10 / Q and above should use the system-as-root + # partition layout, though they may not have the property set. + with self.patch_call( + self.call.device.build_type, + return_value='userdebug'), (self.patch_call( + self.call.device.build_system_root_image, + return_value='')), (self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.Q)), (self.patch_call( + self.call.device.product_name, + return_value='notasailfish')), (self.assertCall( + self.call.adb.Shell('getprop service.adb.root'), + '1\n')): + self.assertTrue(self.device.HasRoot()) + + +class DeviceUtilsEnableRootTest(DeviceUtilsTest): def testEnableRoot_succeeds(self): - with self.assertCalls( - self.call.adb.Root(), - self.call.adb.WaitForDevice(), - (self.call.device.HasRoot(), True)): + with self.assertCalls(self.call.adb.Root(), self.call.adb.WaitForDevice(), + (self.call.device.HasRoot(), True)): self.device.EnableRoot() def testEnableRoot_userBuild(self): - with self.assertCalls( - (self.call.adb.Root(), self.AdbCommandError()), - (self.call.device.IsUserBuild(), True)): + with self.assertCalls((self.call.adb.Root(), self.AdbCommandError()), + (self.call.device.IsUserBuild(), True)): with self.assertRaises(device_errors.CommandFailedError): self.device.EnableRoot() def testEnableRoot_rootFails(self): - with self.assertCalls( - (self.call.adb.Root(), self.AdbCommandError()), - (self.call.device.IsUserBuild(), False)): + with self.assertCalls((self.call.adb.Root(), self.AdbCommandError()), + (self.call.device.IsUserBuild(), False)): with self.assertRaises(device_errors.AdbCommandFailedError): self.device.EnableRoot() @@ -399,14 +469,12 @@ class DeviceUtilsEnableRootTest(DeviceUtilsTest): (self.call.adb.Root(), self.AdbCommandError( output='timeout expired while waiting for device')), - (self.call.device.IsUserBuild(), False), - self.call.adb.WaitForDevice(), + (self.call.device.IsUserBuild(), False), self.call.adb.WaitForDevice(), (self.call.device.HasRoot(), True)): self.device.EnableRoot() class DeviceUtilsIsUserBuildTest(DeviceUtilsTest): - def testIsUserBuild_yes(self): with self.assertCall( self.call.device.GetProp('ro.build.type', cache=True), 'user'): @@ -419,7 +487,6 @@ class DeviceUtilsIsUserBuildTest(DeviceUtilsTest): class DeviceUtilsGetExternalStoragePathTest(DeviceUtilsTest): - def testGetExternalStoragePath_succeeds(self): with self.assertCalls( self.EnsureCacheInitialized(sdcard='/fake/storage/path')): @@ -427,19 +494,64 @@ class DeviceUtilsGetExternalStoragePathTest(DeviceUtilsTest): self.device.GetExternalStoragePath()) def testGetExternalStoragePath_fails(self): + with self.assertCalls(self.EnsureCacheInitialized(sdcard='')): + with self.assertRaises(device_errors.CommandFailedError): + self.device.GetExternalStoragePath() + + +class DeviceUtilsGetAppWritablePathTest(DeviceUtilsTest): + def testGetAppWritablePath_succeeds_sdk_pre_q(self): with self.assertCalls( + (self.call.device.GetProp('ro.build.version.sdk', cache=True), '28'), + self.EnsureCacheInitialized(sdcard='/fake/storage/path')): + self.assertEquals('/fake/storage/path', + self.device.GetAppWritablePath()) + + def testGetAppWritablePath_succeeds_sdk_q(self): + with self.assertCalls( + (self.call.device.GetProp('ro.build.version.sdk', cache=True), '29'), + self.EnsureCacheInitialized(sdcard='/fake/storage/path')): + self.assertEquals('/fake/storage/path/Download', + self.device.GetAppWritablePath()) + + def testGetAppWritablePath_fails(self): + with self.assertCalls( + (self.call.device.GetProp('ro.build.version.sdk', cache=True), '29'), self.EnsureCacheInitialized(sdcard='')): with self.assertRaises(device_errors.CommandFailedError): - self.device.GetExternalStoragePath() + self.device.GetAppWritablePath() + + +class DeviceUtilsIsApplicationInstalledTest(DeviceUtilsTest): + def testIsApplicationInstalled_installed(self): + with self.assertCalls((self.call.device.RunShellCommand( + ['pm', 'list', 'packages', 'some.installed.app'], check_return=True), + ['package:some.installed.app'])): + self.assertTrue(self.device.IsApplicationInstalled('some.installed.app')) + + def testIsApplicationInstalled_notInstalled(self): + with self.assertCalls((self.call.device.RunShellCommand( + ['pm', 'list', 'packages', 'not.installed.app'], check_return=True), + '')): + self.assertFalse(self.device.IsApplicationInstalled('not.installed.app')) + + def testIsApplicationInstalled_substringMatch(self): + with self.assertCalls((self.call.device.RunShellCommand( + ['pm', 'list', 'packages', 'substring.of.package'], check_return=True), + [ + 'package:first.substring.of.package', + 'package:second.substring.of.package', + ])): + self.assertFalse( + self.device.IsApplicationInstalled('substring.of.package')) class DeviceUtilsGetApplicationPathsInternalTest(DeviceUtilsTest): - def testGetApplicationPathsInternal_exists(self): with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '19'), - (self.call.device.RunShellCommand( - ['pm', 'path', 'android'], check_return=True), + (self.call.device.RunShellCommand(['pm', 'path', 'android'], + check_return=True), ['package:/path/to/android.apk'])): self.assertEquals(['/path/to/android.apk'], self.device._GetApplicationPathsInternal('android')) @@ -447,17 +559,16 @@ class DeviceUtilsGetApplicationPathsInternalTest(DeviceUtilsTest): def testGetApplicationPathsInternal_notExists(self): with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '19'), - (self.call.device.RunShellCommand( - ['pm', 'path', 'not.installed.app'], check_return=True), - '')): - self.assertEquals([], - self.device._GetApplicationPathsInternal('not.installed.app')) + (self.call.device.RunShellCommand(['pm', 'path', 'not.installed.app'], + check_return=True), '')): + self.assertEquals( + [], self.device._GetApplicationPathsInternal('not.installed.app')) def testGetApplicationPathsInternal_garbageOutputRaises(self): with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '19'), - (self.call.device.RunShellCommand( - ['pm', 'path', 'android'], check_return=True), + (self.call.device.RunShellCommand(['pm', 'path', 'android'], + check_return=True), ['garbage first line'])): with self.assertRaises(device_errors.CommandFailedError): self.device._GetApplicationPathsInternal('android') @@ -465,24 +576,23 @@ class DeviceUtilsGetApplicationPathsInternalTest(DeviceUtilsTest): def testGetApplicationPathsInternal_outputWarningsIgnored(self): with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '19'), - (self.call.device.RunShellCommand( - ['pm', 'path', 'not.installed.app'], check_return=True), + (self.call.device.RunShellCommand(['pm', 'path', 'not.installed.app'], + check_return=True), ['WARNING: some warning message from pm'])): - self.assertEquals([], - self.device._GetApplicationPathsInternal('not.installed.app')) + self.assertEquals( + [], self.device._GetApplicationPathsInternal('not.installed.app')) def testGetApplicationPathsInternal_fails(self): with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '19'), - (self.call.device.RunShellCommand( - ['pm', 'path', 'android'], check_return=True), + (self.call.device.RunShellCommand(['pm', 'path', 'android'], + check_return=True), self.CommandError('ERROR. Is package manager running?\n'))): with self.assertRaises(device_errors.CommandFailedError): self.device._GetApplicationPathsInternal('android') class DeviceUtils_GetApplicationVersionTest(DeviceUtilsTest): - def test_GetApplicationVersion_exists(self): with self.assertCalls( (self.call.adb.Shell('dumpsys package com.android.chrome'), @@ -511,63 +621,98 @@ class DeviceUtils_GetApplicationVersionTest(DeviceUtilsTest): self.device.GetApplicationVersion('com.android.chrome') -class DeviceUtils_GetPackageArchitectureTest(DeviceUtilsTest): +class DeviceUtils_GetApplicationTargetSdkTest(DeviceUtilsTest): + def test_GetApplicationTargetSdk_exists(self): + with self.assertCalls( + (self.call.device.IsApplicationInstalled('com.android.chrome'), True), + (self.call.device._GetDumpsysOutput(['package', 'com.android.chrome'], + 'targetSdk='), + [' versionCode=413200001 minSdk=21 targetSdk=29'])): + self.assertEquals( + '29', self.device.GetApplicationTargetSdk('com.android.chrome')) + def test_GetApplicationTargetSdk_notExists(self): + with self.assertCalls( + (self.call.device.IsApplicationInstalled('com.android.chrome'), False)): + self.assertIsNone( + self.device.GetApplicationTargetSdk('com.android.chrome')) + + def test_GetApplicationTargetSdk_fails(self): + with self.assertCalls( + (self.call.device.IsApplicationInstalled('com.android.chrome'), True), + (self.call.device._GetDumpsysOutput(['package', 'com.android.chrome'], + 'targetSdk='), [])): + with self.assertRaises(device_errors.CommandFailedError): + self.device.GetApplicationTargetSdk('com.android.chrome') + + def test_GetApplicationTargetSdk_prefinalizedSdk(self): + with self.assertCalls( + (self.call.device.IsApplicationInstalled('com.android.chrome'), True), + (self.call.device._GetDumpsysOutput(['package', 'com.android.chrome'], + 'targetSdk='), + [' versionCode=410301483 minSdk=10000 targetSdk=10000']), + (self.call.device.GetProp('ro.build.version.codename', + cache=True), 'R')): + self.assertEquals( + 'R', self.device.GetApplicationTargetSdk('com.android.chrome')) + + +class DeviceUtils_GetPackageArchitectureTest(DeviceUtilsTest): def test_GetPackageArchitecture_exists(self): with self.assertCall( self.call.device._RunPipedShellCommand( 'dumpsys package com.android.chrome | grep -F primaryCpuAbi'), [' primaryCpuAbi=armeabi-v7a']): self.assertEquals( - abis.ARM, - self.device.GetPackageArchitecture('com.android.chrome')) + abis.ARM, self.device.GetPackageArchitecture('com.android.chrome')) def test_GetPackageArchitecture_notExists(self): with self.assertCall( self.call.device._RunPipedShellCommand( - 'dumpsys package com.android.chrome | grep -F primaryCpuAbi'), - []): + 'dumpsys package com.android.chrome | grep -F primaryCpuAbi'), []): self.assertEquals( - None, - self.device.GetPackageArchitecture('com.android.chrome')) + None, self.device.GetPackageArchitecture('com.android.chrome')) class DeviceUtilsGetApplicationDataDirectoryTest(DeviceUtilsTest): - def testGetApplicationDataDirectory_exists(self): - with self.assertCall( - self.call.device._RunPipedShellCommand( + with self.assertCalls( + (self.call.device.IsApplicationInstalled('foo.bar.baz'), True), + (self.call.device._RunPipedShellCommand( 'pm dump foo.bar.baz | grep dataDir='), - ['dataDir=/data/data/foo.bar.baz']): - self.assertEquals( - '/data/data/foo.bar.baz', - self.device.GetApplicationDataDirectory('foo.bar.baz')) + ['dataDir=/data/data/foo.bar.baz'])): + self.assertEquals('/data/data/foo.bar.baz', + self.device.GetApplicationDataDirectory('foo.bar.baz')) + + def testGetApplicationDataDirectory_notInstalled(self): + with self.assertCalls( + (self.call.device.IsApplicationInstalled('foo.bar.baz'), False)): + with self.assertRaises(device_errors.CommandFailedError): + self.device.GetApplicationDataDirectory('foo.bar.baz') def testGetApplicationDataDirectory_notExists(self): - with self.assertCall( - self.call.device._RunPipedShellCommand( - 'pm dump foo.bar.baz | grep dataDir='), - self.ShellError()): + with self.assertCalls( + (self.call.device.IsApplicationInstalled('foo.bar.baz'), True), + (self.call.device._RunPipedShellCommand( + 'pm dump foo.bar.baz | grep dataDir='), self.ShellError())): with self.assertRaises(device_errors.CommandFailedError): self.device.GetApplicationDataDirectory('foo.bar.baz') @mock.patch('time.sleep', mock.Mock()) class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): - - def testWaitUntilFullyBooted_succeedsNoWifi(self): + def testWaitUntilFullyBooted_succeedsWithDefaults(self): with self.assertCalls( self.call.adb.WaitForDevice(), # sd_card_ready (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), (self.call.adb.Shell('test -d /fake/storage/path'), ''), # pm_ready - (self.call.device._GetApplicationPathsInternal('android', - skip_cache=True), - ['package:/some/fake/path']), + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), ['package:/some/fake/path']), # boot_completed (self.call.device.GetProp('sys.boot_completed', cache=False), '1')): - self.device.WaitUntilFullyBooted(wifi=False) + self.device.WaitUntilFullyBooted(wifi=False, decrypt=False) def testWaitUntilFullyBooted_succeedsWithWifi(self): with self.assertCalls( @@ -576,15 +721,45 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), (self.call.adb.Shell('test -d /fake/storage/path'), ''), # pm_ready - (self.call.device._GetApplicationPathsInternal('android', - skip_cache=True), - ['package:/some/fake/path']), + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), ['package:/some/fake/path']), # boot_completed (self.call.device.GetProp('sys.boot_completed', cache=False), '1'), # wifi_enabled (self.call.adb.Shell('dumpsys wifi'), 'stuff\nWi-Fi is enabled\nmore stuff\n')): - self.device.WaitUntilFullyBooted(wifi=True) + self.device.WaitUntilFullyBooted(wifi=True, decrypt=False) + + def testWaitUntilFullyBooted_succeedsWithDecryptFDE(self): + with self.assertCalls( + self.call.adb.WaitForDevice(), + # sd_card_ready + (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), + (self.call.adb.Shell('test -d /fake/storage/path'), ''), + # pm_ready + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), ['package:/some/fake/path']), + # boot_completed + (self.call.device.GetProp('sys.boot_completed', cache=False), '1'), + # decryption_completed + (self.call.device.GetProp('vold.decrypt', cache=False), + 'trigger_restart_framework')): + self.device.WaitUntilFullyBooted(wifi=False, decrypt=True) + + def testWaitUntilFullyBooted_succeedsWithDecryptNotFDE(self): + with self.assertCalls( + self.call.adb.WaitForDevice(), + # sd_card_ready + (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), + (self.call.adb.Shell('test -d /fake/storage/path'), ''), + # pm_ready + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), ['package:/some/fake/path']), + # boot_completed + (self.call.device.GetProp('sys.boot_completed', cache=False), '1'), + # decryption_completed + (self.call.device.GetProp('vold.decrypt', cache=False), '')): + self.device.WaitUntilFullyBooted(wifi=False, decrypt=True) def testWaitUntilFullyBooted_deviceNotInitiallyAvailable(self): with self.assertCalls( @@ -601,12 +776,11 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), (self.call.adb.Shell('test -d /fake/storage/path'), ''), # pm_ready - (self.call.device._GetApplicationPathsInternal('android', - skip_cache=True), - ['package:/some/fake/path']), + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), ['package:/some/fake/path']), # boot_completed (self.call.device.GetProp('sys.boot_completed', cache=False), '1')): - self.device.WaitUntilFullyBooted(wifi=False) + self.device.WaitUntilFullyBooted(wifi=False, decrypt=False) def testWaitUntilFullyBooted_deviceBrieflyOffline(self): with self.assertCalls( @@ -615,15 +789,14 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), (self.call.adb.Shell('test -d /fake/storage/path'), ''), # pm_ready - (self.call.device._GetApplicationPathsInternal('android', - skip_cache=True), - ['package:/some/fake/path']), + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), ['package:/some/fake/path']), # boot_completed (self.call.device.GetProp('sys.boot_completed', cache=False), self.AdbCommandError()), # boot_completed (self.call.device.GetProp('sys.boot_completed', cache=False), '1')): - self.device.WaitUntilFullyBooted(wifi=False) + self.device.WaitUntilFullyBooted(wifi=False, decrypt=False) def testWaitUntilFullyBooted_sdCardReadyFails_noPath(self): with self.assertCalls( @@ -631,7 +804,7 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): # sd_card_ready (self.call.device.GetExternalStoragePath(), self.CommandError())): with self.assertRaises(device_errors.CommandFailedError): - self.device.WaitUntilFullyBooted(wifi=False) + self.device.WaitUntilFullyBooted(wifi=False, decrypt=False) def testWaitUntilFullyBooted_sdCardReadyFails_notExists(self): with self.assertCalls( @@ -647,7 +820,7 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): (self.call.adb.Shell('test -d /fake/storage/path'), self.TimeoutError())): with self.assertRaises(device_errors.CommandTimeoutError): - self.device.WaitUntilFullyBooted(wifi=False) + self.device.WaitUntilFullyBooted(wifi=False, decrypt=False) def testWaitUntilFullyBooted_devicePmFails(self): with self.assertCalls( @@ -656,19 +829,16 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), (self.call.adb.Shell('test -d /fake/storage/path'), ''), # pm_ready - (self.call.device._GetApplicationPathsInternal('android', - skip_cache=True), - self.CommandError()), + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), self.CommandError()), # pm_ready - (self.call.device._GetApplicationPathsInternal('android', - skip_cache=True), - self.CommandError()), + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), self.CommandError()), # pm_ready - (self.call.device._GetApplicationPathsInternal('android', - skip_cache=True), - self.TimeoutError())): + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), self.TimeoutError())): with self.assertRaises(device_errors.CommandTimeoutError): - self.device.WaitUntilFullyBooted(wifi=False) + self.device.WaitUntilFullyBooted(wifi=False, decrypt=False) def testWaitUntilFullyBooted_bootFails(self): with self.assertCalls( @@ -677,9 +847,8 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), (self.call.adb.Shell('test -d /fake/storage/path'), ''), # pm_ready - (self.call.device._GetApplicationPathsInternal('android', - skip_cache=True), - ['package:/some/fake/path']), + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), ['package:/some/fake/path']), # boot_completed (self.call.device.GetProp('sys.boot_completed', cache=False), '0'), # boot_completed @@ -688,7 +857,7 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): (self.call.device.GetProp('sys.boot_completed', cache=False), self.TimeoutError())): with self.assertRaises(device_errors.CommandTimeoutError): - self.device.WaitUntilFullyBooted(wifi=False) + self.device.WaitUntilFullyBooted(wifi=False, decrypt=False) def testWaitUntilFullyBooted_wifiFails(self): with self.assertCalls( @@ -697,9 +866,8 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), (self.call.adb.Shell('test -d /fake/storage/path'), ''), # pm_ready - (self.call.device._GetApplicationPathsInternal('android', - skip_cache=True), - ['package:/some/fake/path']), + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), ['package:/some/fake/path']), # boot_completed (self.call.device.GetProp('sys.boot_completed', cache=False), '1'), # wifi_enabled @@ -709,331 +877,462 @@ class DeviceUtilsWaitUntilFullyBootedTest(DeviceUtilsTest): # wifi_enabled (self.call.adb.Shell('dumpsys wifi'), self.TimeoutError())): with self.assertRaises(device_errors.CommandTimeoutError): - self.device.WaitUntilFullyBooted(wifi=True) + self.device.WaitUntilFullyBooted(wifi=True, decrypt=False) + + def testWaitUntilFullyBooted_decryptFails(self): + with self.assertCalls( + self.call.adb.WaitForDevice(), + # sd_card_ready + (self.call.device.GetExternalStoragePath(), '/fake/storage/path'), + (self.call.adb.Shell('test -d /fake/storage/path'), ''), + # pm_ready + (self.call.device._GetApplicationPathsInternal( + 'android', skip_cache=True), ['package:/some/fake/path']), + # boot_completed + (self.call.device.GetProp('sys.boot_completed', cache=False), '1'), + # decryption_completed + (self.call.device.GetProp('vold.decrypt', cache=False), + 'trigger_restart_min_framework'), + # decryption_completed + (self.call.device.GetProp('vold.decrypt', cache=False), + 'trigger_restart_min_framework'), + # decryption_completed + (self.call.device.GetProp('vold.decrypt', cache=False), + self.TimeoutError())): + with self.assertRaises(device_errors.CommandTimeoutError): + self.device.WaitUntilFullyBooted(wifi=False, decrypt=True) @mock.patch('time.sleep', mock.Mock()) class DeviceUtilsRebootTest(DeviceUtilsTest): - def testReboot_nonBlocking(self): - with self.assertCalls( - self.call.adb.Reboot(), - (self.call.device.IsOnline(), True), - (self.call.device.IsOnline(), False)): + with self.assertCalls(self.call.adb.Reboot(), + (self.call.device.IsOnline(), True), + (self.call.device.IsOnline(), False)): self.device.Reboot(block=False) def testReboot_blocking(self): with self.assertCalls( - self.call.adb.Reboot(), - (self.call.device.IsOnline(), True), + self.call.adb.Reboot(), (self.call.device.IsOnline(), True), (self.call.device.IsOnline(), False), - self.call.device.WaitUntilFullyBooted(wifi=False)): + self.call.device.WaitUntilFullyBooted(wifi=False, decrypt=False)): self.device.Reboot(block=True) def testReboot_blockUntilWifi(self): with self.assertCalls( - self.call.adb.Reboot(), - (self.call.device.IsOnline(), True), + self.call.adb.Reboot(), (self.call.device.IsOnline(), True), (self.call.device.IsOnline(), False), - self.call.device.WaitUntilFullyBooted(wifi=True)): - self.device.Reboot(block=True, wifi=True) + self.call.device.WaitUntilFullyBooted(wifi=True, decrypt=False)): + self.device.Reboot(block=True, wifi=True, decrypt=False) + + def testReboot_blockUntilDecrypt(self): + with self.assertCalls( + self.call.adb.Reboot(), (self.call.device.IsOnline(), True), + (self.call.device.IsOnline(), False), + self.call.device.WaitUntilFullyBooted(wifi=False, decrypt=True)): + self.device.Reboot(block=True, wifi=False, decrypt=True) class DeviceUtilsInstallTest(DeviceUtilsTest): - mock_apk = _MockApkHelper('/fake/test/app.apk', 'test.package', ['p1']) + mock_apk = _MockApkHelper(TEST_APK_PATH, TEST_PACKAGE, ['p1']) def testInstall_noPriorInstall(self): - with self.patch_call(self.call.device.build_version_sdk, return_value=23): + with self.patch_call( + self.call.device.product_name, + return_value='notflounder'), (self.patch_call( + self.call.device.build_version_sdk, return_value=23)): + with self.assertCalls( + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), []), + self.call.adb.Install(TEST_APK_PATH, + reinstall=False, + streaming=None, + allow_downgrade=False), + (self.call.device.GrantPermissions(TEST_PACKAGE, ['p1']), [])): + self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0) + + def testInstall_noStreaming(self): + with self.patch_call( + self.call.device.product_name, + return_value='flounder'), (self.patch_call( + self.call.device.build_version_sdk, return_value=23)): with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), []), - self.call.adb.Install('/fake/test/app.apk', reinstall=False, + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), []), + self.call.adb.Install(TEST_APK_PATH, + reinstall=False, + streaming=False, allow_downgrade=False), - (self.call.device.GrantPermissions('test.package', ['p1']), [])): + (self.call.device.GrantPermissions(TEST_PACKAGE, ['p1']), [])): self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0) def testInstall_permissionsPreM(self): - with self.patch_call(self.call.device.build_version_sdk, return_value=20): + with self.patch_call( + self.call.device.product_name, + return_value='notflounder'), (self.patch_call( + self.call.device.build_version_sdk, return_value=20)): with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), []), - (self.call.adb.Install('/fake/test/app.apk', reinstall=False, + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), []), + (self.call.adb.Install(TEST_APK_PATH, + reinstall=False, + streaming=None, allow_downgrade=False))): self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0) def testInstall_findPermissions(self): - with self.patch_call(self.call.device.build_version_sdk, return_value=23): + with self.patch_call( + self.call.device.product_name, + return_value='notflounder'), (self.patch_call( + self.call.device.build_version_sdk, return_value=23)): with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), []), - (self.call.adb.Install('/fake/test/app.apk', reinstall=False, + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), []), + (self.call.adb.Install(TEST_APK_PATH, + reinstall=False, + streaming=None, allow_downgrade=False)), - (self.call.device.GrantPermissions('test.package', ['p1']), [])): + (self.call.device.GrantPermissions(TEST_PACKAGE, ['p1']), [])): self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0) def testInstall_passPermissions(self): - with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), []), - (self.call.adb.Install('/fake/test/app.apk', reinstall=False, - allow_downgrade=False)), - (self.call.device.GrantPermissions('test.package', ['p1', 'p2']), [])): - self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0, - permissions=['p1', 'p2']) + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): + with self.assertCalls( + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), []), + (self.call.adb.Install(TEST_APK_PATH, + reinstall=False, + streaming=None, + allow_downgrade=False)), + (self.call.device.GrantPermissions(TEST_PACKAGE, ['p1', 'p2']), [])): + self.device.Install( + DeviceUtilsInstallTest.mock_apk, + retries=0, + permissions=['p1', 'p2']) def testInstall_identicalPriorInstall(self): with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), ['/fake/data/app/test.package.apk']), - (self.call.device._ComputeStaleApks('test.package', - ['/fake/test/app.apk']), - ([], None)), - (self.call.device.ForceStop('test.package'))): - self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0, - permissions=[]) + (self.call.device._ComputeStaleApks(TEST_PACKAGE, [TEST_APK_PATH]), + ([], None)), (self.call.device.ForceStop(TEST_PACKAGE))): + self.device.Install( + DeviceUtilsInstallTest.mock_apk, retries=0, permissions=[]) def testInstall_differentPriorInstall(self): - with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), - ['/fake/data/app/test.package.apk']), - (self.call.device._ComputeStaleApks('test.package', - ['/fake/test/app.apk']), - (['/fake/test/app.apk'], None)), - self.call.device.Uninstall('test.package'), - self.call.adb.Install('/fake/test/app.apk', reinstall=False, - allow_downgrade=False)): - self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0, - permissions=[]) + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): + with self.assertCalls( + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), + ['/fake/data/app/test.package.apk']), + (self.call.device._ComputeStaleApks(TEST_PACKAGE, [TEST_APK_PATH]), + ([TEST_APK_PATH], None)), self.call.device.Uninstall(TEST_PACKAGE), + self.call.adb.Install(TEST_APK_PATH, + reinstall=False, + streaming=None, + allow_downgrade=False)): + self.device.Install( + DeviceUtilsInstallTest.mock_apk, retries=0, permissions=[]) def testInstall_differentPriorInstallSplitApk(self): - with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), - ['/fake/data/app/test.package.apk', - '/fake/data/app/test.package2.apk']), - self.call.device.Uninstall('test.package'), - self.call.adb.Install('/fake/test/app.apk', reinstall=False, - allow_downgrade=False)): - self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0, - permissions=[]) + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): + with self.assertCalls( + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), [ + '/fake/data/app/test.package.apk', + '/fake/data/app/test.package2.apk' + ]), self.call.device.Uninstall(TEST_PACKAGE), + self.call.adb.Install(TEST_APK_PATH, + reinstall=False, + streaming=None, + allow_downgrade=False)): + self.device.Install( + DeviceUtilsInstallTest.mock_apk, retries=0, permissions=[]) def testInstall_differentPriorInstall_reinstall(self): - with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), - ['/fake/data/app/test.package.apk']), - (self.call.device._ComputeStaleApks('test.package', - ['/fake/test/app.apk']), - (['/fake/test/app.apk'], None)), - self.call.adb.Install('/fake/test/app.apk', reinstall=True, - allow_downgrade=False)): - self.device.Install(DeviceUtilsInstallTest.mock_apk, - reinstall=True, retries=0, permissions=[]) + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): + with self.assertCalls( + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), + ['/fake/data/app/test.package.apk']), + (self.call.device._ComputeStaleApks(TEST_PACKAGE, [TEST_APK_PATH]), + ([TEST_APK_PATH], None)), + self.call.adb.Install(TEST_APK_PATH, + reinstall=True, + streaming=None, + allow_downgrade=False)): + self.device.Install( + DeviceUtilsInstallTest.mock_apk, + reinstall=True, + retries=0, + permissions=[]) def testInstall_identicalPriorInstall_reinstall(self): with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), ['/fake/data/app/test.package.apk']), - (self.call.device._ComputeStaleApks('test.package', - ['/fake/test/app.apk']), - ([], None)), - (self.call.device.ForceStop('test.package'))): - self.device.Install(DeviceUtilsInstallTest.mock_apk, - reinstall=True, retries=0, permissions=[]) + (self.call.device._ComputeStaleApks(TEST_PACKAGE, [TEST_APK_PATH]), + ([], None)), (self.call.device.ForceStop(TEST_PACKAGE))): + self.device.Install( + DeviceUtilsInstallTest.mock_apk, + reinstall=True, + retries=0, + permissions=[]) def testInstall_missingApk(self): with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), False)): + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), False)): with self.assertRaises(device_errors.CommandFailedError): self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0) def testInstall_fails(self): - with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), []), - (self.call.adb.Install('/fake/test/app.apk', reinstall=False, - allow_downgrade=False), - self.CommandError('Failure\r\n'))): - with self.assertRaises(device_errors.CommandFailedError): - self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0) + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): + with self.assertCalls( + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), []), + (self.call.adb.Install( + TEST_APK_PATH, + reinstall=False, + streaming=None, + allow_downgrade=False), self.CommandError('Failure\r\n'))): + with self.assertRaises(device_errors.CommandFailedError): + self.device.Install(DeviceUtilsInstallTest.mock_apk, retries=0) def testInstall_downgrade(self): - with self.assertCalls( - (mock.call.os.path.exists('/fake/test/app.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), - ['/fake/data/app/test.package.apk']), - (self.call.device._ComputeStaleApks('test.package', - ['/fake/test/app.apk']), - (['/fake/test/app.apk'], None)), - self.call.adb.Install('/fake/test/app.apk', reinstall=True, - allow_downgrade=True)): - self.device.Install(DeviceUtilsInstallTest.mock_apk, - reinstall=True, retries=0, permissions=[], allow_downgrade=True) + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): + with self.assertCalls( + (self.call.device._FakeInstall(set(), None, 'test.package')), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), + ['/fake/data/app/test.package.apk']), + (self.call.device._ComputeStaleApks(TEST_PACKAGE, [TEST_APK_PATH]), + ([TEST_APK_PATH], None)), + self.call.adb.Install(TEST_APK_PATH, + reinstall=True, + streaming=None, + allow_downgrade=True)): + self.device.Install( + DeviceUtilsInstallTest.mock_apk, + reinstall=True, + retries=0, + permissions=[], + allow_downgrade=True) + + def testInstall_pushesFakeModulesToDevice(self): + @contextlib.contextmanager + def mock_zip_temp_dir(): + yield '/test/tmp/dir' - def testInstall_modulesSpecified(self): - with self.assertRaises(device_errors.CommandFailedError): - self.device.Install(DeviceUtilsInstallTest.mock_apk, - modules=['base']) + mock_apk_with_fake = _MockApkHelper( + TEST_APK_PATH, TEST_PACKAGE, splits=['fake1-master.apk']) + fake_modules = ['fake1'] + with self.patch_call( + self.call.device.product_name, + return_value='notflounder'), (self.patch_call( + self.call.device.build_version_sdk, return_value=23)): + with self.assertCalls( + (mock.call.py_utils.tempfile_ext.NamedTemporaryDirectory(), + mock_zip_temp_dir), + (mock.call.os.rename('fake1-master.apk', '/test/tmp/dir/fake1.apk')), + (self.call.device.PushChangedFiles( + [('/test/tmp/dir', '/data/local/tmp/modules/test.package')], + delete_device_stale=True)), + (mock.call.os.path.exists(TEST_APK_PATH), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), []), + self.call.adb.Install(TEST_APK_PATH, + reinstall=False, + streaming=None, + allow_downgrade=False), + (self.call.device.GrantPermissions(TEST_PACKAGE, None), [])): + self.device.Install( + mock_apk_with_fake, fake_modules=fake_modules, retries=0) class DeviceUtilsInstallSplitApkTest(DeviceUtilsTest): - mock_apk = _MockApkHelper('base.apk', 'test.package', ['p1']) + mock_apk = _MockApkHelper('base.apk', TEST_PACKAGE, ['p1'], + ['split1.apk', 'split2.apk']) def testInstallSplitApk_noPriorInstall(self): - with self.assertCalls( - (self.call.device._CheckSdkLevel(21)), - (mock.call.devil.android.sdk.split_select.SelectSplits( - self.device, 'base.apk', - ['split1.apk', 'split2.apk', 'split3.apk'], - allow_cached_props=False), - ['split2.apk']), - (mock.call.os.path.exists('base.apk'), True), - (mock.call.os.path.exists('split2.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), []), - (self.call.adb.InstallMultiple( - ['base.apk', 'split2.apk'], partial=None, reinstall=False, - allow_downgrade=False))): - self.device.InstallSplitApk(DeviceUtilsInstallSplitApkTest.mock_apk, - ['split1.apk', 'split2.apk', 'split3.apk'], permissions=[], retries=0) + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): + with self.assertCalls( + (mock.call.devil.android.apk_helper.ToSplitHelper( + 'base.apk', ['split1.apk', 'split2.apk']), + DeviceUtilsInstallSplitApkTest.mock_apk), + (self.call.device._CheckSdkLevel(21)), + (mock.call.os.path.exists('base.apk'), True), + (mock.call.os.path.exists('split1.apk'), True), + (mock.call.os.path.exists('split2.apk'), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), []), + (self.call.adb.InstallMultiple( + ['base.apk', 'split1.apk', 'split2.apk'], + partial=None, + reinstall=False, + streaming=None, + allow_downgrade=False))): + self.device.InstallSplitApk( + 'base.apk', ['split1.apk', 'split2.apk'], permissions=[], retries=0) + + def testInstallSplitApk_noStreaming(self): + with self.patch_call( + self.call.device.product_name, return_value='flounder'): + with self.assertCalls( + (mock.call.devil.android.apk_helper.ToSplitHelper( + 'base.apk', ['split1.apk', 'split2.apk']), + DeviceUtilsInstallSplitApkTest.mock_apk), + (self.call.device._CheckSdkLevel(21)), + (mock.call.os.path.exists('base.apk'), True), + (mock.call.os.path.exists('split1.apk'), True), + (mock.call.os.path.exists('split2.apk'), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), []), + (self.call.adb.InstallMultiple( + ['base.apk', 'split1.apk', 'split2.apk'], + partial=None, + reinstall=False, + streaming=False, + allow_downgrade=False))): + self.device.InstallSplitApk( + 'base.apk', ['split1.apk', 'split2.apk'], permissions=[], retries=0) def testInstallSplitApk_partialInstall(self): - with self.assertCalls( - (self.call.device._CheckSdkLevel(21)), - (mock.call.devil.android.sdk.split_select.SelectSplits( - self.device, 'base.apk', - ['split1.apk', 'split2.apk', 'split3.apk'], - allow_cached_props=False), - ['split2.apk']), - (mock.call.os.path.exists('base.apk'), True), - (mock.call.os.path.exists('split2.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), - ['base-on-device.apk', 'split2-on-device.apk']), - (self.call.device._ComputeStaleApks('test.package', - ['base.apk', 'split2.apk']), - (['split2.apk'], None)), - (self.call.adb.InstallMultiple( - ['split2.apk'], partial='test.package', reinstall=True, - allow_downgrade=False))): - self.device.InstallSplitApk(DeviceUtilsInstallSplitApkTest.mock_apk, - ['split1.apk', 'split2.apk', 'split3.apk'], - reinstall=True, permissions=[], retries=0) + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): + with self.assertCalls( + (mock.call.devil.android.apk_helper.ToSplitHelper( + DeviceUtilsInstallSplitApkTest.mock_apk, + ['split1.apk', 'split2.apk']), + DeviceUtilsInstallSplitApkTest.mock_apk), + (self.call.device._CheckSdkLevel(21)), + (mock.call.os.path.exists('base.apk'), True), + (mock.call.os.path.exists('split1.apk'), True), + (mock.call.os.path.exists('split2.apk'), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), + ['base-on-device.apk', 'split2-on-device.apk']), + (self.call.device._ComputeStaleApks( + TEST_PACKAGE, ['base.apk', 'split1.apk', 'split2.apk']), + (['split2.apk'], None)), + (self.call.adb.InstallMultiple(['split2.apk'], + partial=TEST_PACKAGE, + reinstall=True, + streaming=None, + allow_downgrade=False))): + self.device.InstallSplitApk( + DeviceUtilsInstallSplitApkTest.mock_apk, + ['split1.apk', 'split2.apk'], + reinstall=True, + permissions=[], + retries=0) def testInstallSplitApk_downgrade(self): - with self.assertCalls( - (self.call.device._CheckSdkLevel(21)), - (mock.call.devil.android.sdk.split_select.SelectSplits( - self.device, 'base.apk', - ['split1.apk', 'split2.apk', 'split3.apk'], - allow_cached_props=False), - ['split2.apk']), - (mock.call.os.path.exists('base.apk'), True), - (mock.call.os.path.exists('split2.apk'), True), - (self.call.device._GetApplicationPathsInternal('test.package'), - ['base-on-device.apk', 'split2-on-device.apk']), - (self.call.device._ComputeStaleApks('test.package', - ['base.apk', 'split2.apk']), - (['split2.apk'], None)), - (self.call.adb.InstallMultiple( - ['split2.apk'], partial='test.package', reinstall=True, - allow_downgrade=True))): - self.device.InstallSplitApk(DeviceUtilsInstallSplitApkTest.mock_apk, - ['split1.apk', 'split2.apk', 'split3.apk'], - reinstall=True, permissions=[], retries=0, - allow_downgrade=True) + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): + with self.assertCalls( + (mock.call.devil.android.apk_helper.ToSplitHelper( + DeviceUtilsInstallSplitApkTest.mock_apk, + ['split1.apk', 'split2.apk']), + DeviceUtilsInstallSplitApkTest.mock_apk), + (self.call.device._CheckSdkLevel(21)), + (mock.call.os.path.exists('base.apk'), True), + (mock.call.os.path.exists('split1.apk'), True), + (mock.call.os.path.exists('split2.apk'), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), + ['base-on-device.apk', 'split2-on-device.apk']), + (self.call.device._ComputeStaleApks( + TEST_PACKAGE, ['base.apk', 'split1.apk', 'split2.apk']), + (['split2.apk'], None)), + (self.call.adb.InstallMultiple(['split2.apk'], + partial=TEST_PACKAGE, + reinstall=True, + streaming=None, + allow_downgrade=True))): + self.device.InstallSplitApk( + DeviceUtilsInstallSplitApkTest.mock_apk, + ['split1.apk', 'split2.apk'], + reinstall=True, + permissions=[], + retries=0, + allow_downgrade=True) def testInstallSplitApk_missingSplit(self): with self.assertCalls( + (mock.call.devil.android.apk_helper.ToSplitHelper( + DeviceUtilsInstallSplitApkTest.mock_apk, + ['split1.apk', 'split2.apk']), + DeviceUtilsInstallSplitApkTest.mock_apk), (self.call.device._CheckSdkLevel(21)), - (mock.call.devil.android.sdk.split_select.SelectSplits( - self.device, 'base.apk', - ['split1.apk', 'split2.apk', 'split3.apk'], - allow_cached_props=False), - ['split2.apk']), (mock.call.os.path.exists('base.apk'), True), - (mock.call.os.path.exists('split2.apk'), False)): - with self.assertRaises(device_errors.CommandFailedError): - self.device.InstallSplitApk(DeviceUtilsInstallSplitApkTest.mock_apk, - ['split1.apk', 'split2.apk', 'split3.apk'], permissions=[], - retries=0) + (mock.call.os.path.exists('split1.apk'), True), + (mock.call.os.path.exists('split2.apk'), False)),\ + self.assertRaises(device_errors.CommandFailedError): + self.device.InstallSplitApk( + DeviceUtilsInstallSplitApkTest.mock_apk, ['split1.apk', 'split2.apk'], + permissions=[], + retries=0) def testInstallSplitApk_previouslyNonSplit(self): - with self.assertCalls( - (self.call.device._CheckSdkLevel(21)), - (mock.call.devil.android.sdk.split_select.SelectSplits( - self.device, 'base.apk', - ['split1.apk', 'split2.apk', 'split3.apk'], - allow_cached_props=False), - ['split2.apk']), - (mock.call.os.path.exists('base.apk'), True), - (mock.call.os.path.exists('split2.apk'), True), - (self.call.device._GetApplicationPathsInternal( - 'test.package'), ['/fake/data/app/test.package.apk']), - self.call.device.Uninstall('test.package'), - (self.call.adb.InstallMultiple( - ['base.apk', 'split2.apk'], partial=None, reinstall=False, - allow_downgrade=False))): - self.device.InstallSplitApk(DeviceUtilsInstallSplitApkTest.mock_apk, - ['split1.apk', 'split2.apk', 'split3.apk'], permissions=[], retries=0) - - -class DeviceUtilsInstallBundleTest(DeviceUtilsTest): - mock_apk = _MockApkHelper('/fake/test/app_bundle', 'test.package', ['p1']) - - def testInstallBundle_noPriorInstall(self): - with self.patch_call(self.call.device.build_version_sdk, return_value=23): - with self.assertCalls( - (mock.call.devil.utils.cmd_helper.RunCmd( - ['/fake/test/app_bundle', 'install', '--device', - self.device.serial]), 0), - (self.call.device.GrantPermissions('test.package', ['p1']), [])): - self.device.Install(DeviceUtilsInstallBundleTest.mock_apk) - - def testInstallBundle_modulesSpecified(self): - with self.patch_call(self.call.device.build_version_sdk, return_value=23): - with self.assertCalls( - (mock.call.devil.utils.cmd_helper.RunCmd( - ['/fake/test/app_bundle', 'install', '--device', - self.device.serial, '-m', 'base']), 0), - (self.call.device.GrantPermissions('test.package', ['p1']), [])): - self.device.Install( - DeviceUtilsInstallBundleTest.mock_apk, modules=['base']) - - def testInstallBundle_permissionsPreM(self): - with self.patch_call(self.call.device.build_version_sdk, return_value=20): + with self.patch_call( + self.call.device.product_name, return_value='notflounder'): with self.assertCalls( - (mock.call.devil.utils.cmd_helper.RunCmd( - ['/fake/test/app_bundle', 'install', '--device', - self.device.serial]), 0)): - self.device.Install(DeviceUtilsInstallBundleTest.mock_apk) - - def testInstallBundle_splitApks(self): - with self.assertRaises(device_errors.CommandFailedError): - self.device.InstallSplitApk( - DeviceUtilsInstallBundleTest.mock_apk, ['apk1', 'apk2']) + (mock.call.devil.android.apk_helper.ToSplitHelper( + DeviceUtilsInstallSplitApkTest.mock_apk, + ['split1.apk', 'split2.apk']), + DeviceUtilsInstallSplitApkTest.mock_apk), + (self.call.device._CheckSdkLevel(21)), + (mock.call.os.path.exists('base.apk'), True), + (mock.call.os.path.exists('split1.apk'), True), + (mock.call.os.path.exists('split2.apk'), True), + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), + ['/fake/data/app/test.package.apk']), + self.call.device.Uninstall(TEST_PACKAGE), + (self.call.adb.InstallMultiple( + ['base.apk', 'split1.apk', 'split2.apk'], + partial=None, + reinstall=False, + streaming=None, + allow_downgrade=False))): + self.device.InstallSplitApk( + DeviceUtilsInstallSplitApkTest.mock_apk, + ['split1.apk', 'split2.apk'], + permissions=[], + retries=0) class DeviceUtilsUninstallTest(DeviceUtilsTest): - def testUninstall_callsThrough(self): with self.assertCalls( - (self.call.device._GetApplicationPathsInternal('test.package'), - ['/path.apk']), - self.call.adb.Uninstall('test.package', True)): - self.device.Uninstall('test.package', True) + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), + ['/path.apk']), self.call.adb.Uninstall(TEST_PACKAGE, True)): + self.device.Uninstall(TEST_PACKAGE, True) def testUninstall_noop(self): with self.assertCalls( - (self.call.device._GetApplicationPathsInternal('test.package'), [])): - self.device.Uninstall('test.package', True) + (self.call.device._GetApplicationPathsInternal(TEST_PACKAGE), [])): + self.device.Uninstall(TEST_PACKAGE, True) class DeviceUtilsSuTest(DeviceUtilsTest): - def testSu_preM(self): with self.patch_call( self.call.device.build_version_sdk, @@ -1048,82 +1347,87 @@ class DeviceUtilsSuTest(DeviceUtilsTest): class DeviceUtilsRunShellCommandTest(DeviceUtilsTest): - def setUp(self): super(DeviceUtilsRunShellCommandTest, self).setUp() self.device.NeedsSU = mock.Mock(return_value=False) def testRunShellCommand_commandAsList(self): with self.assertCall(self.call.adb.Shell('pm list packages'), ''): - self.device.RunShellCommand( - ['pm', 'list', 'packages'], check_return=True) + self.device.RunShellCommand(['pm', 'list', 'packages'], check_return=True) def testRunShellCommand_commandAsListQuoted(self): with self.assertCall(self.call.adb.Shell("echo 'hello world' '$10'"), ''): - self.device.RunShellCommand( - ['echo', 'hello world', '$10'], check_return=True) + self.device.RunShellCommand(['echo', 'hello world', '$10'], + check_return=True) def testRunShellCommand_commandAsString(self): with self.assertCall(self.call.adb.Shell('echo "$VAR"'), ''): - self.device.RunShellCommand( - 'echo "$VAR"', shell=True, check_return=True) + self.device.RunShellCommand('echo "$VAR"', shell=True, check_return=True) def testNewRunShellImpl_withEnv(self): with self.assertCall( self.call.adb.Shell('VAR=some_string echo "$VAR"'), ''): self.device.RunShellCommand( - 'echo "$VAR"', shell=True, check_return=True, + 'echo "$VAR"', + shell=True, + check_return=True, env={'VAR': 'some_string'}) def testNewRunShellImpl_withEnvQuoted(self): with self.assertCall( self.call.adb.Shell('PATH="$PATH:/other/path" run_this'), ''): - self.device.RunShellCommand( - ['run_this'], check_return=True, env={'PATH': '$PATH:/other/path'}) + self.device.RunShellCommand(['run_this'], + check_return=True, + env={'PATH': '$PATH:/other/path'}) def testNewRunShellImpl_withEnv_failure(self): with self.assertRaises(KeyError): - self.device.RunShellCommand( - ['some_cmd'], check_return=True, env={'INVALID NAME': 'value'}) + self.device.RunShellCommand(['some_cmd'], + check_return=True, + env={'INVALID NAME': 'value'}) def testNewRunShellImpl_withCwd(self): with self.assertCall(self.call.adb.Shell('cd /some/test/path && ls'), ''): - self.device.RunShellCommand( - ['ls'], check_return=True, cwd='/some/test/path') + self.device.RunShellCommand(['ls'], + check_return=True, + cwd='/some/test/path') def testNewRunShellImpl_withCwdQuoted(self): with self.assertCall( self.call.adb.Shell("cd '/some test/path with/spaces' && ls"), ''): - self.device.RunShellCommand( - ['ls'], check_return=True, cwd='/some test/path with/spaces') + self.device.RunShellCommand(['ls'], + check_return=True, + cwd='/some test/path with/spaces') def testRunShellCommand_withHugeCmd(self): payload = 'hi! ' * 1024 expected_cmd = "echo '%s'" % payload with self.assertCalls( - (mock.call.devil.android.device_temp_file.DeviceTempFile( - self.adb, suffix='.sh'), MockTempFile('/sdcard/temp-123.sh')), - self.call.device._WriteFileWithPush('/sdcard/temp-123.sh', expected_cmd), - (self.call.adb.Shell('sh /sdcard/temp-123.sh'), payload + '\n')): - self.assertEquals( - [payload], - self.device.RunShellCommand(['echo', payload], check_return=True)) + (mock.call.devil.android.device_temp_file.DeviceTempFile( + self.adb, suffix='.sh'), MockTempFile('/sdcard/temp-123.sh')), + self.call.device._WriteFileWithPush('/sdcard/temp-123.sh', + expected_cmd), + (self.call.adb.Shell('sh /sdcard/temp-123.sh'), payload + '\n')): + self.assertEquals([payload], + self.device.RunShellCommand(['echo', payload], + check_return=True)) def testRunShellCommand_withHugeCmdAndSu(self): payload = 'hi! ' * 1024 expected_cmd_without_su = """sh -c 'echo '"'"'%s'"'"''""" % payload expected_cmd = 'su -c %s' % expected_cmd_without_su with self.assertCalls( - (self.call.device.NeedsSU(), True), - (self.call.device._Su(expected_cmd_without_su), expected_cmd), - (mock.call.devil.android.device_temp_file.DeviceTempFile( - self.adb, suffix='.sh'), MockTempFile('/sdcard/temp-123.sh')), - self.call.device._WriteFileWithPush('/sdcard/temp-123.sh', expected_cmd), - (self.call.adb.Shell('sh /sdcard/temp-123.sh'), payload + '\n')): - self.assertEquals( - [payload], - self.device.RunShellCommand( - ['echo', payload], check_return=True, as_root=True)) + (self.call.device.NeedsSU(), True), + (self.call.device._Su(expected_cmd_without_su), expected_cmd), + (mock.call.devil.android.device_temp_file.DeviceTempFile( + self.adb, suffix='.sh'), MockTempFile('/sdcard/temp-123.sh')), + self.call.device._WriteFileWithPush('/sdcard/temp-123.sh', + expected_cmd), + (self.call.adb.Shell('sh /sdcard/temp-123.sh'), payload + '\n')): + self.assertEquals([payload], + self.device.RunShellCommand(['echo', payload], + check_return=True, + as_root=True)) def testRunShellCommand_withSu(self): expected_cmd_without_su = "sh -c 'setprop service.adb.root 0'" @@ -1132,18 +1436,18 @@ class DeviceUtilsRunShellCommandTest(DeviceUtilsTest): (self.call.device.NeedsSU(), True), (self.call.device._Su(expected_cmd_without_su), expected_cmd), (self.call.adb.Shell(expected_cmd), '')): - self.device.RunShellCommand( - ['setprop', 'service.adb.root', '0'], - check_return=True, as_root=True) + self.device.RunShellCommand(['setprop', 'service.adb.root', '0'], + check_return=True, + as_root=True) def testRunShellCommand_withRunAs(self): expected_cmd_without_run_as = "sh -c 'mkdir -p files'" expected_cmd = ( 'run-as org.devil.test_package %s' % expected_cmd_without_run_as) with self.assertCall(self.call.adb.Shell(expected_cmd), ''): - self.device.RunShellCommand( - ['mkdir', '-p', 'files'], - check_return=True, run_as='org.devil.test_package') + self.device.RunShellCommand(['mkdir', '-p', 'files'], + check_return=True, + run_as='org.devil.test_package') def testRunShellCommand_withRunAsAndSu(self): expected_cmd_with_nothing = "sh -c 'mkdir -p files'" @@ -1156,17 +1460,17 @@ class DeviceUtilsRunShellCommandTest(DeviceUtilsTest): (self.call.device.NeedsSU(), True), (self.call.device._Su(expected_cmd_without_su), expected_cmd), (self.call.adb.Shell(expected_cmd), '')): - self.device.RunShellCommand( - ['mkdir', '-p', 'files'], - check_return=True, run_as='org.devil.test_package', - as_root=True) + self.device.RunShellCommand(['mkdir', '-p', 'files'], + check_return=True, + run_as='org.devil.test_package', + as_root=True) def testRunShellCommand_manyLines(self): cmd = 'ls /some/path' with self.assertCall(self.call.adb.Shell(cmd), 'file1\nfile2\nfile3\n'): - self.assertEquals( - ['file1', 'file2', 'file3'], - self.device.RunShellCommand(cmd.split(), check_return=True)) + self.assertEquals(['file1', 'file2', 'file3'], + self.device.RunShellCommand( + cmd.split(), check_return=True)) def testRunShellCommand_manyLinesRawOutput(self): cmd = 'ls /some/path' @@ -1210,8 +1514,8 @@ class DeviceUtilsRunShellCommandTest(DeviceUtilsTest): def testRunShellCommand_singleLine_failTooManyLines(self): cmd = 'echo $VALUE' - with self.assertCall(self.call.adb.Shell(cmd), - 'some value\nanother value\n'): + with self.assertCall( + self.call.adb.Shell(cmd), 'some value\nanother value\n'): with self.assertRaises(device_errors.CommandFailedError): self.device.RunShellCommand( cmd, shell=True, check_return=True, single_line=True) @@ -1220,9 +1524,9 @@ class DeviceUtilsRunShellCommandTest(DeviceUtilsTest): cmd = 'echo $ANDROID_DATA' output = '/data\n' with self.assertCall(self.call.adb.Shell(cmd), output): - self.assertEquals( - [output.rstrip()], - self.device.RunShellCommand(cmd, shell=True, check_return=True)) + self.assertEquals([output.rstrip()], + self.device.RunShellCommand( + cmd, shell=True, check_return=True)) def testRunShellCommand_checkReturn_failure(self): cmd = 'ls /root' @@ -1235,9 +1539,9 @@ class DeviceUtilsRunShellCommandTest(DeviceUtilsTest): cmd = 'ls /root' output = 'opendir failed, Permission denied\n' with self.assertCall(self.call.adb.Shell(cmd), self.ShellError(output)): - self.assertEquals( - [output.rstrip()], - self.device.RunShellCommand(cmd.split(), check_return=False)) + self.assertEquals([output.rstrip()], + self.device.RunShellCommand( + cmd.split(), check_return=False)) def testRunShellCommand_largeOutput_enabled(self): cmd = 'echo $VALUE' @@ -1245,14 +1549,15 @@ class DeviceUtilsRunShellCommandTest(DeviceUtilsTest): cmd_redirect = '( %s )>%s 2>&1' % (cmd, temp_file.name) with self.assertCalls( (mock.call.devil.android.device_temp_file.DeviceTempFile(self.adb), - temp_file), - (self.call.adb.Shell(cmd_redirect)), - (self.call.device.ReadFile(temp_file.name, force_pull=True), - 'something')): - self.assertEquals( - ['something'], - self.device.RunShellCommand( - cmd, shell=True, large_output=True, check_return=True)) + temp_file), + (self.call.adb.Shell(cmd_redirect)), (self.call.device.ReadFile( + temp_file.name, force_pull=True), 'something')): + self.assertEquals(['something'], + self.device.RunShellCommand( + cmd, + shell=True, + large_output=True, + check_return=True)) def testRunShellCommand_largeOutput_disabledNoTrigger(self): cmd = 'something' @@ -1267,23 +1572,20 @@ class DeviceUtilsRunShellCommandTest(DeviceUtilsTest): with self.assertCalls( (self.call.adb.Shell(cmd), self.ShellError('', None)), (mock.call.devil.android.device_temp_file.DeviceTempFile(self.adb), - temp_file), - (self.call.adb.Shell(cmd_redirect)), - (self.call.device.ReadFile(mock.ANY, force_pull=True), - 'something')): - self.assertEquals( - ['something'], - self.device.RunShellCommand(cmd, shell=True, check_return=True)) + temp_file), (self.call.adb.Shell(cmd_redirect)), + (self.call.device.ReadFile(mock.ANY, force_pull=True), 'something')): + self.assertEquals(['something'], + self.device.RunShellCommand( + cmd, shell=True, check_return=True)) class DeviceUtilsRunPipedShellCommandTest(DeviceUtilsTest): - def testRunPipedShellCommand_success(self): with self.assertCall( self.call.device.RunShellCommand( 'ps | grep foo; echo "PIPESTATUS: ${PIPESTATUS[@]}"', - shell=True, check_return=True), - ['This line contains foo', 'PIPESTATUS: 0 0']): + shell=True, + check_return=True), ['This line contains foo', 'PIPESTATUS: 0 0']): self.assertEquals(['This line contains foo'], self.device._RunPipedShellCommand('ps | grep foo')) @@ -1291,8 +1593,8 @@ class DeviceUtilsRunPipedShellCommandTest(DeviceUtilsTest): with self.assertCall( self.call.device.RunShellCommand( 'ps | grep foo; echo "PIPESTATUS: ${PIPESTATUS[@]}"', - shell=True, check_return=True), - ['PIPESTATUS: 1 0']): + shell=True, + check_return=True), ['PIPESTATUS: 1 0']): with self.assertRaises(device_errors.AdbShellCommandFailedError) as ec: self.device._RunPipedShellCommand('ps | grep foo') self.assertEquals([1, 0], ec.exception.status) @@ -1301,8 +1603,8 @@ class DeviceUtilsRunPipedShellCommandTest(DeviceUtilsTest): with self.assertCall( self.call.device.RunShellCommand( 'ps | grep foo; echo "PIPESTATUS: ${PIPESTATUS[@]}"', - shell=True, check_return=True), - ['PIPESTATUS: 0 1']): + shell=True, + check_return=True), ['PIPESTATUS: 0 1']): with self.assertRaises(device_errors.AdbShellCommandFailedError) as ec: self.device._RunPipedShellCommand('ps | grep foo') self.assertEquals([0, 1], ec.exception.status) @@ -1311,8 +1613,8 @@ class DeviceUtilsRunPipedShellCommandTest(DeviceUtilsTest): with self.assertCall( self.call.device.RunShellCommand( 'ps | grep foo; echo "PIPESTATUS: ${PIPESTATUS[@]}"', - shell=True, check_return=True), - ['foo.bar'] * 256 + ['foo.ba']): + shell=True, + check_return=True), ['foo.bar'] * 256 + ['foo.ba']): with self.assertRaises(device_errors.AdbShellCommandFailedError) as ec: self.device._RunPipedShellCommand('ps | grep foo') self.assertIs(None, ec.exception.status) @@ -1320,7 +1622,6 @@ class DeviceUtilsRunPipedShellCommandTest(DeviceUtilsTest): @mock.patch('time.sleep', mock.Mock()) class DeviceUtilsKillAllTest(DeviceUtilsTest): - def testKillAll_noMatchingProcessesFailure(self): with self.assertCall(self.call.device.ListProcesses('test_process'), []): with self.assertRaises(device_errors.CommandFailedError): @@ -1331,12 +1632,11 @@ class DeviceUtilsKillAllTest(DeviceUtilsTest): self.assertEqual(0, self.device.KillAll('test_process', quiet=True)) def testKillAll_nonblocking(self): - with self.assertCalls( - (self.call.device.ListProcesses('some.process'), - Processes(('some.process', 1234), ('some.process.thing', 5678))), - (self.call.adb.Shell('kill -9 1234 5678'), '')): - self.assertEquals( - 2, self.device.KillAll('some.process', blocking=False)) + with self.assertCalls((self.call.device.ListProcesses('some.process'), + Processes(('some.process', 1234), + ('some.process.thing', 5678))), + (self.call.adb.Shell('kill -9 1234 5678'), '')): + self.assertEquals(2, self.device.KillAll('some.process', blocking=False)) def testKillAll_blocking(self): with self.assertCalls( @@ -1345,62 +1645,59 @@ class DeviceUtilsKillAllTest(DeviceUtilsTest): (self.call.adb.Shell('kill -9 1234 5678'), ''), (self.call.device.ListProcesses('some.process'), Processes(('some.process.thing', 5678))), - (self.call.device.ListProcesses('some.process'), - # Other instance with different pid. - Processes(('some.process', 111)))): - self.assertEquals( - 2, self.device.KillAll('some.process', blocking=True)) + ( + self.call.device.ListProcesses('some.process'), + # Other instance with different pid. + Processes(('some.process', 111)))): + self.assertEquals(2, self.device.KillAll('some.process', blocking=True)) def testKillAll_exactNonblocking(self): - with self.assertCalls( - (self.call.device.ListProcesses('some.process'), - Processes(('some.process', 1234), ('some.process.thing', 5678))), - (self.call.adb.Shell('kill -9 1234'), '')): + with self.assertCalls((self.call.device.ListProcesses('some.process'), + Processes(('some.process', 1234), + ('some.process.thing', 5678))), + (self.call.adb.Shell('kill -9 1234'), '')): self.assertEquals( 1, self.device.KillAll('some.process', exact=True, blocking=False)) def testKillAll_exactBlocking(self): - with self.assertCalls( - (self.call.device.ListProcesses('some.process'), - Processes(('some.process', 1234), ('some.process.thing', 5678))), - (self.call.adb.Shell('kill -9 1234'), ''), - (self.call.device.ListProcesses('some.process'), - Processes(('some.process', 1234), ('some.process.thing', 5678))), - (self.call.device.ListProcesses('some.process'), - Processes(('some.process.thing', 5678)))): + with self.assertCalls((self.call.device.ListProcesses('some.process'), + Processes(('some.process', 1234), + ('some.process.thing', 5678))), + (self.call.adb.Shell('kill -9 1234'), ''), + (self.call.device.ListProcesses('some.process'), + Processes(('some.process', 1234), + ('some.process.thing', 5678))), + (self.call.device.ListProcesses('some.process'), + Processes(('some.process.thing', 5678)))): self.assertEquals( 1, self.device.KillAll('some.process', exact=True, blocking=True)) def testKillAll_root(self): with self.assertCalls( (self.call.device.ListProcesses('some.process'), - Processes(('some.process', 1234))), - (self.call.device.NeedsSU(), True), + Processes(('some.process', 1234))), (self.call.device.NeedsSU(), True), (self.call.device._Su("sh -c 'kill -9 1234'"), "su -c sh -c 'kill -9 1234'"), (self.call.adb.Shell("su -c sh -c 'kill -9 1234'"), '')): - self.assertEquals( - 1, self.device.KillAll('some.process', as_root=True)) + self.assertEquals(1, self.device.KillAll('some.process', as_root=True)) def testKillAll_sigterm(self): - with self.assertCalls( - (self.call.device.ListProcesses('some.process'), - Processes(('some.process', 1234))), - (self.call.adb.Shell('kill -15 1234'), '')): + with self.assertCalls((self.call.device.ListProcesses('some.process'), + Processes(('some.process', 1234))), + (self.call.adb.Shell('kill -15 1234'), '')): self.assertEquals( 1, self.device.KillAll('some.process', signum=device_signal.SIGTERM)) def testKillAll_multipleInstances(self): - with self.assertCalls( - (self.call.device.ListProcesses('some.process'), - Processes(('some.process', 1234), ('some.process', 4567))), - (self.call.adb.Shell('kill -15 1234 4567'), '')): + with self.assertCalls((self.call.device.ListProcesses('some.process'), + Processes(('some.process', 1234), + ('some.process', 4567))), + (self.call.adb.Shell('kill -15 1234 4567'), '')): self.assertEquals( 2, self.device.KillAll('some.process', signum=device_signal.SIGTERM)) class DeviceUtilsStartActivityTest(DeviceUtilsTest): - def testStartActivity_actionOnly(self): test_intent = intent.Intent(action='android.intent.action.VIEW') with self.assertCall( @@ -1410,9 +1707,10 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent) def testStartActivity_success(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main') + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main') with self.assertCall( self.call.adb.Shell('am start ' '-a android.intent.action.VIEW ' @@ -1421,9 +1719,10 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent) def testStartActivity_failure(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main') + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main') with self.assertCall( self.call.adb.Shell('am start ' '-a android.intent.action.VIEW ' @@ -1433,9 +1732,10 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent) def testStartActivity_blocking(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main') + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main') with self.assertCall( self.call.adb.Shell('am start ' '-W ' @@ -1445,10 +1745,11 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent, blocking=True) def testStartActivity_withCategory(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main', - category='android.intent.category.HOME') + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main', + category='android.intent.category.HOME') with self.assertCall( self.call.adb.Shell('am start ' '-a android.intent.action.VIEW ' @@ -1458,11 +1759,13 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent) def testStartActivity_withMultipleCategories(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main', - category=['android.intent.category.HOME', - 'android.intent.category.BROWSABLE']) + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main', + category=[ + 'android.intent.category.HOME', 'android.intent.category.BROWSABLE' + ]) with self.assertCall( self.call.adb.Shell('am start ' '-a android.intent.action.VIEW ' @@ -1473,10 +1776,11 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent) def testStartActivity_withData(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main', - data='http://www.google.com/') + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main', + data='http://www.google.com/') with self.assertCall( self.call.adb.Shell('am start ' '-a android.intent.action.VIEW ' @@ -1486,10 +1790,11 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent) def testStartActivity_withStringExtra(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main', - extras={'foo': 'test'}) + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main', + extras={'foo': 'test'}) with self.assertCall( self.call.adb.Shell('am start ' '-a android.intent.action.VIEW ' @@ -1499,10 +1804,11 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent) def testStartActivity_withBoolExtra(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main', - extras={'foo': True}) + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main', + extras={'foo': True}) with self.assertCall( self.call.adb.Shell('am start ' '-a android.intent.action.VIEW ' @@ -1512,10 +1818,11 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent) def testStartActivity_withIntExtra(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main', - extras={'foo': 123}) + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main', + extras={'foo': 123}) with self.assertCall( self.call.adb.Shell('am start ' '-a android.intent.action.VIEW ' @@ -1525,22 +1832,24 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent) def testStartActivity_withTraceFile(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main') + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main') with self.assertCall( self.call.adb.Shell('am start ' '--start-profiler test_trace_file.out ' '-a android.intent.action.VIEW ' '-n test.package/.Main'), 'Starting: Intent { act=android.intent.action.VIEW }'): - self.device.StartActivity(test_intent, - trace_file_name='test_trace_file.out') + self.device.StartActivity( + test_intent, trace_file_name='test_trace_file.out') def testStartActivity_withForceStop(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main') + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main') with self.assertCall( self.call.adb.Shell('am start ' '-S ' @@ -1550,13 +1859,14 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): self.device.StartActivity(test_intent, force_stop=True) def testStartActivity_withFlags(self): - test_intent = intent.Intent(action='android.intent.action.VIEW', - package='test.package', - activity='.Main', - flags=[ - intent.FLAG_ACTIVITY_NEW_TASK, - intent.FLAG_ACTIVITY_RESET_TASK_IF_NEEDED - ]) + test_intent = intent.Intent( + action='android.intent.action.VIEW', + package=TEST_PACKAGE, + activity='.Main', + flags=[ + intent.FLAG_ACTIVITY_NEW_TASK, + intent.FLAG_ACTIVITY_RESET_TASK_IF_NEEDED + ]) with self.assertCall( self.call.adb.Shell('am start ' '-a android.intent.action.VIEW ' @@ -1568,11 +1878,12 @@ class DeviceUtilsStartActivityTest(DeviceUtilsTest): class DeviceUtilsStartServiceTest(DeviceUtilsTest): def testStartService_success(self): - test_intent = intent.Intent(action='android.intent.action.START', - package='test.package', - activity='.Main') - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.NOUGAT): + test_intent = intent.Intent( + action='android.intent.action.START', + package=TEST_PACKAGE, + activity='.Main') + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.NOUGAT): with self.assertCall( self.call.adb.Shell('am startservice ' '-a android.intent.action.START ' @@ -1581,11 +1892,12 @@ class DeviceUtilsStartServiceTest(DeviceUtilsTest): self.device.StartService(test_intent) def testStartService_failure(self): - test_intent = intent.Intent(action='android.intent.action.START', - package='test.package', - activity='.Main') - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.NOUGAT): + test_intent = intent.Intent( + action='android.intent.action.START', + package=TEST_PACKAGE, + activity='.Main') + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.NOUGAT): with self.assertCall( self.call.adb.Shell('am startservice ' '-a android.intent.action.START ' @@ -1595,11 +1907,12 @@ class DeviceUtilsStartServiceTest(DeviceUtilsTest): self.device.StartService(test_intent) def testStartService_withUser(self): - test_intent = intent.Intent(action='android.intent.action.START', - package='test.package', - activity='.Main') - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.NOUGAT): + test_intent = intent.Intent( + action='android.intent.action.START', + package=TEST_PACKAGE, + activity='.Main') + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.NOUGAT): with self.assertCall( self.call.adb.Shell('am startservice ' '--user TestUser ' @@ -1609,11 +1922,12 @@ class DeviceUtilsStartServiceTest(DeviceUtilsTest): self.device.StartService(test_intent, user_id='TestUser') def testStartService_onOreo(self): - test_intent = intent.Intent(action='android.intent.action.START', - package='test.package', - activity='.Main') - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.OREO): + test_intent = intent.Intent( + action='android.intent.action.START', + package=TEST_PACKAGE, + activity='.Main') + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.OREO): with self.assertCall( self.call.adb.Shell('am start-service ' '-a android.intent.action.START ' @@ -1623,50 +1937,64 @@ class DeviceUtilsStartServiceTest(DeviceUtilsTest): class DeviceUtilsStartInstrumentationTest(DeviceUtilsTest): - def testStartInstrumentation_nothing(self): with self.assertCalls( self.call.device.RunShellCommand( 'p=test.package;am instrument "$p"/.TestInstrumentation', - shell=True, check_return=True, large_output=True)): + shell=True, + check_return=True, + large_output=True)): self.device.StartInstrumentation( 'test.package/.TestInstrumentation', - finish=False, raw=False, extras=None) + finish=False, + raw=False, + extras=None) def testStartInstrumentation_finish(self): - with self.assertCalls( - (self.call.device.RunShellCommand( - 'p=test.package;am instrument -w "$p"/.TestInstrumentation', - shell=True, check_return=True, large_output=True), - ['OK (1 test)'])): + with self.assertCalls((self.call.device.RunShellCommand( + 'p=test.package;am instrument -w "$p"/.TestInstrumentation', + shell=True, + check_return=True, + large_output=True), ['OK (1 test)'])): output = self.device.StartInstrumentation( 'test.package/.TestInstrumentation', - finish=True, raw=False, extras=None) + finish=True, + raw=False, + extras=None) self.assertEquals(['OK (1 test)'], output) def testStartInstrumentation_raw(self): with self.assertCalls( self.call.device.RunShellCommand( 'p=test.package;am instrument -r "$p"/.TestInstrumentation', - shell=True, check_return=True, large_output=True)): + shell=True, + check_return=True, + large_output=True)): self.device.StartInstrumentation( 'test.package/.TestInstrumentation', - finish=False, raw=True, extras=None) + finish=False, + raw=True, + extras=None) def testStartInstrumentation_extras(self): with self.assertCalls( self.call.device.RunShellCommand( 'p=test.package;am instrument -e "$p".foo Foo -e bar \'Val \'"$p" ' '"$p"/.TestInstrumentation', - shell=True, check_return=True, large_output=True)): + shell=True, + check_return=True, + large_output=True)): self.device.StartInstrumentation( 'test.package/.TestInstrumentation', - finish=False, raw=False, extras={'test.package.foo': 'Foo', - 'bar': 'Val test.package'}) + finish=False, + raw=False, + extras={ + 'test.package.foo': 'Foo', + 'bar': 'Val test.package' + }) class DeviceUtilsBroadcastIntentTest(DeviceUtilsTest): - def testBroadcastIntent_noExtras(self): test_intent = intent.Intent(action='test.package.with.an.INTENT') with self.assertCall( @@ -1675,8 +2003,8 @@ class DeviceUtilsBroadcastIntentTest(DeviceUtilsTest): self.device.BroadcastIntent(test_intent) def testBroadcastIntent_withExtra(self): - test_intent = intent.Intent(action='test.package.with.an.INTENT', - extras={'foo': 'bar value'}) + test_intent = intent.Intent( + action='test.package.with.an.INTENT', extras={'foo': 'bar value'}) with self.assertCall( self.call.adb.Shell( "am broadcast -a test.package.with.an.INTENT --es foo 'bar value'"), @@ -1684,8 +2012,8 @@ class DeviceUtilsBroadcastIntentTest(DeviceUtilsTest): self.device.BroadcastIntent(test_intent) def testBroadcastIntent_withExtra_noValue(self): - test_intent = intent.Intent(action='test.package.with.an.INTENT', - extras={'foo': None}) + test_intent = intent.Intent( + action='test.package.with.an.INTENT', extras={'foo': None}) with self.assertCall( self.call.adb.Shell( 'am broadcast -a test.package.with.an.INTENT --esn foo'), @@ -1694,137 +2022,136 @@ class DeviceUtilsBroadcastIntentTest(DeviceUtilsTest): class DeviceUtilsGoHomeTest(DeviceUtilsTest): - def testGoHome_popupsExist(self): with self.assertCalls( - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), []), - (self.call.device.RunShellCommand( - ['am', 'start', '-W', '-a', 'android.intent.action.MAIN', - '-c', 'android.intent.category.HOME'], check_return=True), - 'Starting: Intent { act=android.intent.action.MAIN }\r\n'''), - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), []), - (self.call.device.RunShellCommand( - ['input', 'keyevent', '66'], check_return=True)), - (self.call.device.RunShellCommand( - ['input', 'keyevent', '4'], check_return=True)), - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), []), + (self.call.device.RunShellCommand([ + 'am', 'start', '-W', '-a', 'android.intent.action.MAIN', '-c', + 'android.intent.category.HOME' + ], + check_return=True), + 'Starting: Intent { act=android.intent.action.MAIN }\r\n' + ''), + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), []), + (self.call.device.RunShellCommand(['input', 'keyevent', '66'], + check_return=True)), + (self.call.device.RunShellCommand(['input', 'keyevent', '4'], + check_return=True)), + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), ['mCurrentFocus Launcher'])): self.device.GoHome() def testGoHome_willRetry(self): with self.assertCalls( + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), []), + (self.call.device.RunShellCommand([ + 'am', 'start', '-W', '-a', 'android.intent.action.MAIN', '-c', + 'android.intent.category.HOME' + ], + check_return=True), + 'Starting: Intent { act=android.intent.action.MAIN }\r\n' + ''), + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), []), (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), []), - (self.call.device.RunShellCommand( - ['am', 'start', '-W', '-a', 'android.intent.action.MAIN', - '-c', 'android.intent.category.HOME'], check_return=True), - 'Starting: Intent { act=android.intent.action.MAIN }\r\n'''), - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), []), - (self.call.device.RunShellCommand( - ['input', 'keyevent', '66'], check_return=True,)), - (self.call.device.RunShellCommand( - ['input', 'keyevent', '4'], check_return=True)), - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), []), - (self.call.device.RunShellCommand( - ['input', 'keyevent', '66'], check_return=True)), - (self.call.device.RunShellCommand( - ['input', 'keyevent', '4'], check_return=True)), - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), + ['input', 'keyevent', '66'], + check_return=True, + )), (self.call.device.RunShellCommand(['input', 'keyevent', '4'], + check_return=True)), + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), []), + (self.call.device.RunShellCommand(['input', 'keyevent', '66'], + check_return=True)), + (self.call.device.RunShellCommand(['input', 'keyevent', '4'], + check_return=True)), + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), self.TimeoutError())): with self.assertRaises(device_errors.CommandTimeoutError): self.device.GoHome() def testGoHome_alreadyFocused(self): with self.assertCall( - self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), + self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), ['mCurrentFocus Launcher']): self.device.GoHome() def testGoHome_alreadyFocusedAlternateCase(self): with self.assertCall( - self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), + self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), [' mCurrentFocus .launcher/.']): self.device.GoHome() def testGoHome_obtainsFocusAfterGoingHome(self): with self.assertCalls( - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), []), - (self.call.device.RunShellCommand( - ['am', 'start', '-W', '-a', 'android.intent.action.MAIN', - '-c', 'android.intent.category.HOME'], check_return=True), - 'Starting: Intent { act=android.intent.action.MAIN }\r\n'''), - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), - ['mCurrentFocus Launcher'])): + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), []), + (self.call.device.RunShellCommand([ + 'am', 'start', '-W', '-a', 'android.intent.action.MAIN', '-c', + 'android.intent.category.HOME' + ], + check_return=True), + 'Starting: Intent { act=android.intent.action.MAIN }\r\n' + ''), (self.call.device.RunShellCommand( + ['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), ['mCurrentFocus Launcher'])): self.device.GoHome() class DeviceUtilsForceStopTest(DeviceUtilsTest): - def testForceStop(self): with self.assertCalls( - (self.call.device.GetApplicationPids('test.package'), [1111]), - (self.call.device.RunShellCommand( - ['am', 'force-stop', 'test.package'], - check_return=True), - ['Success'])): - self.device.ForceStop('test.package') + (self.call.device.GetApplicationPids(TEST_PACKAGE), [1111]), + (self.call.device.RunShellCommand(['am', 'force-stop', TEST_PACKAGE], + check_return=True), ['Success'])): + self.device.ForceStop(TEST_PACKAGE) def testForceStop_NoProcessFound(self): - with self.assertCall( - self.call.device.GetApplicationPids('test.package'), []): - self.device.ForceStop('test.package') + with self.assertCall(self.call.device.GetApplicationPids(TEST_PACKAGE), []): + self.device.ForceStop(TEST_PACKAGE) class DeviceUtilsClearApplicationStateTest(DeviceUtilsTest): - def testClearApplicationState_setPermissions(self): with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '17'), (self.call.device._GetApplicationPathsInternal('this.package.exists'), ['/data/app/this.package.exists.apk']), (self.call.device.RunShellCommand( - ['pm', 'clear', 'this.package.exists'], - check_return=True), + ['pm', 'clear', 'this.package.exists'], check_return=True), ['Success']), - (self.call.device.GrantPermissions( - 'this.package.exists', ['p1']), [])): + (self.call.device.GrantPermissions('this.package.exists', ['p1']), [])): self.device.ClearApplicationState( 'this.package.exists', permissions=['p1']) def testClearApplicationState_packageDoesntExist(self): with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '11'), - (self.call.device._GetApplicationPathsInternal('does.not.exist'), - [])): + (self.call.device._GetApplicationPathsInternal('does.not.exist'), [])): self.device.ClearApplicationState('does.not.exist') def testClearApplicationState_packageDoesntExistOnAndroidJBMR2OrAbove(self): with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '18'), (self.call.device.RunShellCommand( - ['pm', 'clear', 'this.package.does.not.exist'], - check_return=True), + ['pm', 'clear', 'this.package.does.not.exist'], check_return=True), ['Failed'])): self.device.ClearApplicationState('this.package.does.not.exist') @@ -1834,8 +2161,7 @@ class DeviceUtilsClearApplicationStateTest(DeviceUtilsTest): (self.call.device._GetApplicationPathsInternal('this.package.exists'), ['/data/app/this.package.exists.apk']), (self.call.device.RunShellCommand( - ['pm', 'clear', 'this.package.exists'], - check_return=True), + ['pm', 'clear', 'this.package.exists'], check_return=True), ['Success'])): self.device.ClearApplicationState('this.package.exists') @@ -1843,21 +2169,18 @@ class DeviceUtilsClearApplicationStateTest(DeviceUtilsTest): with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '18'), (self.call.device.RunShellCommand( - ['pm', 'clear', 'this.package.exists'], - check_return=True), + ['pm', 'clear', 'this.package.exists'], check_return=True), ['Success'])): self.device.ClearApplicationState('this.package.exists') class DeviceUtilsSendKeyEventTest(DeviceUtilsTest): - def testSendKeyEvent(self): with self.assertCall(self.call.adb.Shell('input keyevent 66'), ''): self.device.SendKeyEvent(66) class DeviceUtilsPushChangedFilesIndividuallyTest(DeviceUtilsTest): - def testPushChangedFilesIndividually_empty(self): test_files = [] with self.assertCalls(): @@ -1869,23 +2192,19 @@ class DeviceUtilsPushChangedFilesIndividuallyTest(DeviceUtilsTest): self.device._PushChangedFilesIndividually(test_files) def testPushChangedFilesIndividually_multiple(self): - test_files = [ - ('/test/host/path/file1', '/test/device/path/file1'), - ('/test/host/path/file2', '/test/device/path/file2')] + test_files = [('/test/host/path/file1', '/test/device/path/file1'), + ('/test/host/path/file2', '/test/device/path/file2')] with self.assertCalls( - self.call.adb.Push(*test_files[0]), - self.call.adb.Push(*test_files[1])): + self.call.adb.Push(*test_files[0]), self.call.adb.Push(*test_files[1])): self.device._PushChangedFilesIndividually(test_files) class DeviceUtilsPushChangedFilesZippedTest(DeviceUtilsTest): - def testPushChangedFilesZipped_noUnzipCommand(self): test_files = [('/test/host/path/file1', '/test/device/path/file1')] - with self.assertCalls( - (self.call.device._MaybeInstallCommands(), False)): - self.assertFalse(self.device._PushChangedFilesZipped(test_files, - ['/test/dir'])) + with self.assertCalls((self.call.device._MaybeInstallCommands(), False)): + self.assertFalse( + self.device._PushChangedFilesZipped(test_files, ['/test/dir'])) def _testPushChangedFilesZipped_spec(self, test_files): @contextlib.contextmanager @@ -1895,28 +2214,25 @@ class DeviceUtilsPushChangedFilesZippedTest(DeviceUtilsTest): with self.assertCalls( (self.call.device._MaybeInstallCommands(), True), (mock.call.py_utils.tempfile_ext.NamedTemporaryDirectory(), - mock_zip_temp_dir), - (mock.call.devil.utils.zip_utils.WriteZipFile( - '/test/temp/dir/tmp.zip', test_files)), - (mock.call.os.path.getsize( - '/test/temp/dir/tmp.zip'), 123), + mock_zip_temp_dir), (mock.call.devil.utils.zip_utils.WriteZipFile( + '/test/temp/dir/tmp.zip', test_files)), + (mock.call.os.path.getsize('/test/temp/dir/tmp.zip'), 123), (self.call.device.NeedsSU(), True), - (mock.call.devil.android.device_temp_file.DeviceTempFile(self.adb, - suffix='.zip'), - MockTempFile('/test/sdcard/foo123.zip')), - self.call.adb.Push( - '/test/temp/dir/tmp.zip', '/test/sdcard/foo123.zip'), + (mock.call.devil.android.device_temp_file.DeviceTempFile( + self.adb, suffix='.zip'), MockTempFile('/test/sdcard/foo123.zip')), + self.call.adb.Push('/test/temp/dir/tmp.zip', '/test/sdcard/foo123.zip'), self.call.device.RunShellCommand( 'unzip /test/sdcard/foo123.zip&&chmod -R 777 /test/dir', - shell=True, as_root=True, + shell=True, + as_root=True, env={'PATH': '/data/local/tmp/bin:$PATH'}, check_return=True)): - self.assertTrue(self.device._PushChangedFilesZipped(test_files, - ['/test/dir'])) + self.assertTrue( + self.device._PushChangedFilesZipped(test_files, ['/test/dir'])) def testPushChangedFilesZipped_single(self): - self._testPushChangedFilesZipped_spec( - [('/test/host/path/file1', '/test/device/path/file1')]) + self._testPushChangedFilesZipped_spec([('/test/host/path/file1', + '/test/device/path/file1')]) def testPushChangedFilesZipped_multiple(self): self._testPushChangedFilesZipped_spec( @@ -1925,37 +2241,42 @@ class DeviceUtilsPushChangedFilesZippedTest(DeviceUtilsTest): class DeviceUtilsPathExistsTest(DeviceUtilsTest): - def testPathExists_pathExists(self): with self.assertCall( - self.call.device.RunShellCommand( - ['test', '-e', '/path/file exists'], - as_root=False, check_return=True, timeout=10, retries=0), - []): + self.call.device.RunShellCommand(['test', '-e', '/path/file exists'], + as_root=False, + check_return=True, + timeout=10, + retries=0), []): self.assertTrue(self.device.PathExists('/path/file exists')) def testPathExists_multiplePathExists(self): with self.assertCall( self.call.device.RunShellCommand( ['test', '-e', '/path 1', '-a', '-e', '/path2'], - as_root=False, check_return=True, timeout=10, retries=0), - []): + as_root=False, + check_return=True, + timeout=10, + retries=0), []): self.assertTrue(self.device.PathExists(('/path 1', '/path2'))) def testPathExists_pathDoesntExist(self): with self.assertCall( self.call.device.RunShellCommand( ['test', '-e', '/path/file.not.exists'], - as_root=False, check_return=True, timeout=10, retries=0), - self.ShellError()): + as_root=False, + check_return=True, + timeout=10, + retries=0), self.ShellError()): self.assertFalse(self.device.PathExists('/path/file.not.exists')) def testPathExists_asRoot(self): with self.assertCall( - self.call.device.RunShellCommand( - ['test', '-e', '/root/path/exists'], - as_root=True, check_return=True, timeout=10, retries=0), - self.ShellError()): + self.call.device.RunShellCommand(['test', '-e', '/root/path/exists'], + as_root=True, + check_return=True, + timeout=10, + retries=0), self.ShellError()): self.assertFalse( self.device.PathExists('/root/path/exists', as_root=True)) @@ -1963,52 +2284,51 @@ class DeviceUtilsPathExistsTest(DeviceUtilsTest): with self.assertCall( self.call.device.RunShellCommand( ['test', '-e', '/path/file.not.exists'], - as_root=False, check_return=True, timeout=10, retries=0), - self.ShellError()): + as_root=False, + check_return=True, + timeout=10, + retries=0), self.ShellError()): self.assertFalse(self.device.FileExists('/path/file.not.exists')) class DeviceUtilsRemovePathTest(DeviceUtilsTest): - def testRemovePath_regular(self): with self.assertCall( - self.call.device.RunShellCommand( - ['rm', 'some file'], as_root=False, check_return=True), - []): + self.call.device.RunShellCommand(['rm', 'some file'], + as_root=False, + check_return=True), []): self.device.RemovePath('some file') def testRemovePath_withForce(self): with self.assertCall( - self.call.device.RunShellCommand( - ['rm', '-f', 'some file'], as_root=False, check_return=True), - []): + self.call.device.RunShellCommand(['rm', '-f', 'some file'], + as_root=False, + check_return=True), []): self.device.RemovePath('some file', force=True) def testRemovePath_recursively(self): with self.assertCall( - self.call.device.RunShellCommand( - ['rm', '-r', '/remove/this/dir'], as_root=False, check_return=True), - []): + self.call.device.RunShellCommand(['rm', '-r', '/remove/this/dir'], + as_root=False, + check_return=True), []): self.device.RemovePath('/remove/this/dir', recursive=True) def testRemovePath_withRoot(self): with self.assertCall( - self.call.device.RunShellCommand( - ['rm', 'some file'], as_root=True, check_return=True), - []): + self.call.device.RunShellCommand(['rm', 'some file'], + as_root=True, + check_return=True), []): self.device.RemovePath('some file', as_root=True) def testRemovePath_manyPaths(self): with self.assertCall( - self.call.device.RunShellCommand( - ['rm', 'eeny', 'meeny', 'miny', 'moe'], - as_root=False, check_return=True), - []): + self.call.device.RunShellCommand(['rm', 'eeny', 'meeny', 'miny', 'moe'], + as_root=False, + check_return=True), []): self.device.RemovePath(['eeny', 'meeny', 'miny', 'moe']) class DeviceUtilsPullFileTest(DeviceUtilsTest): - def testPullFile_existsOnDevice(self): with mock.patch('os.path.exists', return_value=True): with self.assertCall( @@ -2030,33 +2350,35 @@ class DeviceUtilsPullFileTest(DeviceUtilsTest): def testPullFile_asRoot(self): with mock.patch('os.path.exists', return_value=True): with self.assertCalls( - (self.call.device.NeedsSU(), True), - (self.call.device.PathExists('/this/file/can.be.read.with.su', - as_root=True), True), + (self.call.device.NeedsSU(), True), (self.call.device.PathExists( + '/this/file/can.be.read.with.su', as_root=True), True), (mock.call.devil.android.device_temp_file.DeviceTempFile(self.adb), MockTempFile('/sdcard/tmp/on.device')), self.call.device.RunShellCommand( 'SRC=/this/file/can.be.read.with.su DEST=/sdcard/tmp/on.device;' 'cp "$SRC" "$DEST" && chmod 666 "$DEST"', - shell=True, as_root=True, check_return=True), - (self.call.adb.Pull('/sdcard/tmp/on.device', - '/test/file/host/path'))): - self.device.PullFile('/this/file/can.be.read.with.su', - '/test/file/host/path', as_root=True) + shell=True, + as_root=True, + check_return=True), (self.call.adb.Pull('/sdcard/tmp/on.device', + '/test/file/host/path'))): + self.device.PullFile( + '/this/file/can.be.read.with.su', + '/test/file/host/path', + as_root=True) def testPullFile_asRootDoesntExistOnDevice(self): with mock.patch('os.path.exists', return_value=True): with self.assertCalls( - (self.call.device.NeedsSU(), True), - (self.call.device.PathExists('/data/app/test.file.does.not.exist', - as_root=True), False)): + (self.call.device.NeedsSU(), True), (self.call.device.PathExists( + '/data/app/test.file.does.not.exist', as_root=True), False)): with self.assertRaises(device_errors.CommandFailedError): - self.device.PullFile('/data/app/test.file.does.not.exist', - '/test/file/host/path', as_root=True) + self.device.PullFile( + '/data/app/test.file.does.not.exist', + '/test/file/host/path', + as_root=True) class DeviceUtilsReadFileTest(DeviceUtilsTest): - def testReadFileWithPull_success(self): tmp_host_dir = '/tmp/dir/on.host/' tmp_host = MockTempFile('/tmp/dir/on.host/tmp_ReadFileWithPull') @@ -2073,93 +2395,82 @@ class DeviceUtilsReadFileTest(DeviceUtilsTest): def testReadFileWithPull_rejected(self): tmp_host_dir = '/tmp/dir/on.host/' - with self.assertCalls( - (mock.call.tempfile.mkdtemp(), tmp_host_dir), - (self.call.adb.Pull('/path/to/device/file', mock.ANY), - self.CommandError()), - (mock.call.os.path.exists(tmp_host_dir), True), - (mock.call.shutil.rmtree(tmp_host_dir), None)): + with self.assertCalls((mock.call.tempfile.mkdtemp(), tmp_host_dir), + (self.call.adb.Pull('/path/to/device/file', mock.ANY), + self.CommandError()), + (mock.call.os.path.exists(tmp_host_dir), True), + (mock.call.shutil.rmtree(tmp_host_dir), None)): with self.assertRaises(device_errors.CommandFailedError): self.device._ReadFileWithPull('/path/to/device/file') - def testReadFile_exists(self): - with self.assertCalls( - (self.call.device.FileSize('/read/this/test/file', as_root=False), 256), - (self.call.device.RunShellCommand( - ['cat', '/read/this/test/file'], - as_root=False, check_return=True), - ['this is a test file'])): - self.assertEqual('this is a test file\n', - self.device.ReadFile('/read/this/test/file')) - - def testReadFile_exists2(self): - # Same as testReadFile_exists, but uses Android N ls output. - with self.assertCalls( - (self.call.device.FileSize('/read/this/test/file', as_root=False), 256), - (self.call.device.RunShellCommand( - ['cat', '/read/this/test/file'], - as_root=False, check_return=True), - ['this is a test file'])): - self.assertEqual('this is a test file\n', - self.device.ReadFile('/read/this/test/file')) - - def testReadFile_doesNotExist(self): - with self.assertCall( - self.call.device.FileSize('/this/file/does.not.exist', as_root=False), - self.CommandError('File does not exist')): - with self.assertRaises(device_errors.CommandFailedError): - self.device.ReadFile('/this/file/does.not.exist') - - def testReadFile_zeroSize(self): + def testReadFile_withSU_zeroSize(self): with self.assertCalls( - (self.call.device.FileSize('/this/file/has/zero/size', as_root=False), - 0), - (self.call.device._ReadFileWithPull('/this/file/has/zero/size'), + (self.call.device.NeedsSU(), True), + (self.call.device.FileSize( + '/this/file/has/zero/size', as_root=True), 0), + (mock.call.devil.android.device_temp_file.DeviceTempFile(self.adb), + MockTempFile('/sdcard/tmp/on.device')), + self.call.device.RunShellCommand( + 'SRC=/this/file/has/zero/size DEST=/sdcard/tmp/on.device;' + 'cp "$SRC" "$DEST" && chmod 666 "$DEST"', + shell=True, + as_root=True, + check_return=True), + (self.call.device._ReadFileWithPull('/sdcard/tmp/on.device'), 'but it has contents\n')): self.assertEqual('but it has contents\n', - self.device.ReadFile('/this/file/has/zero/size')) + self.device.ReadFile('/this/file/has/zero/size', + as_root=True)) def testReadFile_withSU(self): with self.assertCalls( + (self.call.device.NeedsSU(), True), (self.call.device.FileSize( '/this/file/can.be.read.with.su', as_root=True), 256), (self.call.device.RunShellCommand( ['cat', '/this/file/can.be.read.with.su'], - as_root=True, check_return=True), - ['this is a test file', 'read with su'])): + as_root=True, + check_return=True), ['this is a test file', 'read with su'])): self.assertEqual( 'this is a test file\nread with su\n', - self.device.ReadFile('/this/file/can.be.read.with.su', - as_root=True)) + self.device.ReadFile('/this/file/can.be.read.with.su', as_root=True)) + + def testReadFile_withSU_doesNotExist(self): + with self.assertCalls( + (self.call.device.NeedsSU(), True), + (self.call.device.FileSize('/this/file/does.not.exist', as_root=True), + self.CommandError('File does not exist'))): + with self.assertRaises(device_errors.CommandFailedError): + self.device.ReadFile('/this/file/does.not.exist', as_root=True) def testReadFile_withPull(self): contents = 'a' * 123456 with self.assertCalls( - (self.call.device.FileSize('/read/this/big/test/file', as_root=False), - 123456), (self.call.device._ReadFileWithPull('/read/this/big/test/file'), contents)): - self.assertEqual( - contents, self.device.ReadFile('/read/this/big/test/file')) + self.assertEqual(contents, + self.device.ReadFile('/read/this/big/test/file')) def testReadFile_withPullAndSU(self): contents = 'b' * 123456 with self.assertCalls( + (self.call.device.NeedsSU(), True), (self.call.device.FileSize( '/this/big/file/can.be.read.with.su', as_root=True), 123456), - (self.call.device.NeedsSU(), True), (mock.call.devil.android.device_temp_file.DeviceTempFile(self.adb), MockTempFile('/sdcard/tmp/on.device')), self.call.device.RunShellCommand( 'SRC=/this/big/file/can.be.read.with.su DEST=/sdcard/tmp/on.device;' 'cp "$SRC" "$DEST" && chmod 666 "$DEST"', - shell=True, as_root=True, check_return=True), + shell=True, + as_root=True, + check_return=True), (self.call.device._ReadFileWithPull('/sdcard/tmp/on.device'), contents)): self.assertEqual( contents, - self.device.ReadFile('/this/big/file/can.be.read.with.su', - as_root=True)) + self.device.ReadFile( + '/this/big/file/can.be.read.with.su', as_root=True)) def testReadFile_forcePull(self): contents = 'a' * 123456 @@ -2172,13 +2483,12 @@ class DeviceUtilsReadFileTest(DeviceUtilsTest): class DeviceUtilsWriteFileTest(DeviceUtilsTest): - def testWriteFileWithPush_success(self): tmp_host = MockTempFile('/tmp/file/on.host') contents = 'some interesting contents' - with self.assertCalls( - (mock.call.tempfile.NamedTemporaryFile(), tmp_host), - self.call.adb.Push('/tmp/file/on.host', '/path/to/device/file')): + with self.assertCalls((mock.call.tempfile.NamedTemporaryFile(), tmp_host), + self.call.adb.Push('/tmp/file/on.host', + '/path/to/device/file')): self.device._WriteFileWithPush('/path/to/device/file', contents) tmp_host.file.write.assert_called_once_with(contents) @@ -2213,17 +2523,19 @@ class DeviceUtilsWriteFileTest(DeviceUtilsTest): self.call.device._WriteFileWithPush('/sdcard/tmp/on.device', contents), self.call.device.RunShellCommand( ['cp', '/sdcard/tmp/on.device', '/path/to/device/file'], - as_root=True, check_return=True)): + as_root=True, + check_return=True)): self.device.WriteFile('/path/to/device/file', contents, as_root=True) def testWriteFile_withEcho(self): - with self.assertCall(self.call.adb.Shell( - "echo -n the.contents > /test/file/to.write"), ''): + with self.assertCall( + self.call.adb.Shell("echo -n the.contents > /test/file/to.write"), ''): self.device.WriteFile('/test/file/to.write', 'the.contents') def testWriteFile_withEchoAndQuotes(self): - with self.assertCall(self.call.adb.Shell( - "echo -n 'the contents' > '/test/file/to write'"), ''): + with self.assertCall( + self.call.adb.Shell("echo -n 'the contents' > '/test/file/to write'"), + ''): self.device.WriteFile('/test/file/to write', 'the contents') def testWriteFile_withEchoAndSU(self): @@ -2232,8 +2544,7 @@ class DeviceUtilsWriteFileTest(DeviceUtilsTest): with self.assertCalls( (self.call.device.NeedsSU(), True), (self.call.device._Su(expected_cmd_without_su), expected_cmd), - (self.call.adb.Shell(expected_cmd), - '')): + (self.call.adb.Shell(expected_cmd), '')): self.device.WriteFile('/test/file', 'contents', as_root=True) @@ -2241,42 +2552,45 @@ class DeviceUtilsStatDirectoryTest(DeviceUtilsTest): # Note: Also tests ListDirectory in testStatDirectory_fileList. EXAMPLE_LS_OUTPUT = [ - 'total 12345', - 'drwxr-xr-x 19 root root 0 1970-04-06 18:03 .', - 'drwxr-xr-x 19 root root 0 1970-04-06 18:03 ..', - 'drwxr-xr-x 6 root root 1970-01-01 00:00 some_dir', - '-rw-r--r-- 1 root root 723 1971-01-01 07:04 some_file', - '-rw-r----- 1 root root 327 2009-02-13 23:30 My Music File', - # Some Android versions escape spaces in file names - '-rw-rw-rw- 1 root root 0 2018-01-11 13:35 Local\\ State', - # Older Android versions do not print st_nlink - 'lrwxrwxrwx root root 1970-01-01 00:00 lnk -> /some/path', - 'srwxrwx--- system system 2016-05-31 17:25 a_socket1', - 'drwxrwxrwt system misc 1970-11-23 02:25 tmp', - 'drwxr-s--- system shell 1970-11-23 02:24 my_cmd', - 'cr--r----- root system 10, 183 1971-01-01 07:04 random', - 'brw------- root root 7, 0 1971-01-01 07:04 block_dev', - '-rwS------ root shell 157404 2015-04-13 15:44 silly', + 'total 12345', + 'drwxr-xr-x 19 root root 0 1970-04-06 18:03 .', + 'drwxr-xr-x 19 root root 0 1970-04-06 18:03 ..', + 'drwxr-xr-x 6 root root 1970-01-01 00:00 some_dir', + '-rw-r--r-- 1 root root 723 1971-01-01 07:04 some_file', + '-rw-r----- 1 root root 327 2009-02-13 23:30 My Music File', + # Some Android versions escape spaces in file names + '-rw-rw-rw- 1 root root 0 2018-01-11 13:35 Local\\ State', + # Older Android versions do not print st_nlink + 'lrwxrwxrwx root root 1970-01-01 00:00 lnk -> /a/path', + 'srwxrwx--- system system 2016-05-31 17:25 a_socket1', + 'drwxrwxrwt system misc 1970-11-23 02:25 tmp', + 'drwxr-s--- system shell 1970-11-23 02:24 my_cmd', + 'cr--r----- root system 10, 183 1971-01-01 07:04 random', + 'brw------- root root 7, 0 1971-01-01 07:04 block_dev', + '-rwS------ root shell 157404 2015-04-13 15:44 silly', ] FILENAMES = [ - 'some_dir', 'some_file', 'My Music File', 'Local State', 'lnk', - 'a_socket1', 'tmp', 'my_cmd', 'random', 'block_dev', 'silly'] + 'some_dir', 'some_file', 'My Music File', 'Local State', 'lnk', + 'a_socket1', 'tmp', 'my_cmd', 'random', 'block_dev', 'silly' + ] def getStatEntries(self, path_given='/', path_listed='/'): with self.assertCall( - self.call.device.RunShellCommand( - ['ls', '-a', '-l', path_listed], - check_return=True, as_root=False, env={'TZ': 'utc'}), + self.call.device.RunShellCommand(['ls', '-a', '-l', path_listed], + check_return=True, + as_root=False, + env={'TZ': 'utc'}), self.EXAMPLE_LS_OUTPUT): entries = self.device.StatDirectory(path_given) return {f['filename']: f for f in entries} def getListEntries(self): with self.assertCall( - self.call.device.RunShellCommand( - ['ls', '-a', '-l', '/'], - check_return=True, as_root=False, env={'TZ': 'utc'}), + self.call.device.RunShellCommand(['ls', '-a', '-l', '/'], + check_return=True, + as_root=False, + env={'TZ': 'utc'}), self.EXAMPLE_LS_OUTPUT): return self.device.ListDirectory('/') @@ -2290,12 +2604,12 @@ class DeviceUtilsStatDirectoryTest(DeviceUtilsTest): def testStatDirectory_fileModes(self): expected_modes = ( - ('some_dir', stat.S_ISDIR), - ('some_file', stat.S_ISREG), - ('lnk', stat.S_ISLNK), - ('a_socket1', stat.S_ISSOCK), - ('block_dev', stat.S_ISBLK), - ('random', stat.S_ISCHR), + ('some_dir', stat.S_ISDIR), + ('some_file', stat.S_ISREG), + ('lnk', stat.S_ISLNK), + ('a_socket1', stat.S_ISSOCK), + ('block_dev', stat.S_ISBLK), + ('random', stat.S_ISCHR), ) entries = self.getStatEntries() for filename, check in expected_modes: @@ -2303,16 +2617,16 @@ class DeviceUtilsStatDirectoryTest(DeviceUtilsTest): def testStatDirectory_filePermissions(self): should_have = ( - ('some_file', stat.S_IWUSR), # Owner can write. - ('tmp', stat.S_IXOTH), # Others can execute. - ('tmp', stat.S_ISVTX), # Has sticky bit. - ('my_cmd', stat.S_ISGID), # Has set-group-ID bit. - ('silly', stat.S_ISUID), # Has set UID bit. + ('some_file', stat.S_IWUSR), # Owner can write. + ('tmp', stat.S_IXOTH), # Others can execute. + ('tmp', stat.S_ISVTX), # Has sticky bit. + ('my_cmd', stat.S_ISGID), # Has set-group-ID bit. + ('silly', stat.S_ISUID), # Has set UID bit. ) should_not_have = ( - ('some_file', stat.S_IWOTH), # Others can't write. - ('block_dev', stat.S_IRGRP), # Group can't read. - ('silly', stat.S_IXUSR), # Owner can't execute. + ('some_file', stat.S_IWOTH), # Others can't write. + ('block_dev', stat.S_IRGRP), # Group can't read. + ('silly', stat.S_IXUSR), # Owner can't execute. ) entries = self.getStatEntries() for filename, bit in should_have: @@ -2353,17 +2667,21 @@ class DeviceUtilsStatDirectoryTest(DeviceUtilsTest): def testStatDirectory_symbolicLinks(self): entries = self.getStatEntries() - self.assertEqual(entries['lnk']['symbolic_link_to'], '/some/path') + self.assertEqual(entries['lnk']['symbolic_link_to'], '/a/path') for d in entries.itervalues(): self.assertEqual('symbolic_link_to' in d, stat.S_ISLNK(d['st_mode'])) class DeviceUtilsStatPathTest(DeviceUtilsTest): - EXAMPLE_DIRECTORY = [ - {'filename': 'foo.txt', 'st_size': 123, 'st_time': 456}, - {'filename': 'some_dir', 'st_time': 0} - ] + EXAMPLE_DIRECTORY = [{ + 'filename': 'foo.txt', + 'st_size': 123, + 'st_time': 456 + }, { + 'filename': 'some_dir', + 'st_time': 0 + }] INDEX = {e['filename']: e for e in EXAMPLE_DIRECTORY} def testStatPath_file(self): @@ -2397,17 +2715,20 @@ class DeviceUtilsStatPathTest(DeviceUtilsTest): class DeviceUtilsFileSizeTest(DeviceUtilsTest): - EXAMPLE_DIRECTORY = [ - {'filename': 'foo.txt', 'st_size': 123, 'st_mtime': 456}, - {'filename': 'some_dir', 'st_mtime': 0} - ] + EXAMPLE_DIRECTORY = [{ + 'filename': 'foo.txt', + 'st_size': 123, + 'st_mtime': 456 + }, { + 'filename': 'some_dir', + 'st_mtime': 0 + }] def testFileSize_file(self): with self.assertCall( self.call.device.StatDirectory('/data/local/tmp', as_root=False), self.EXAMPLE_DIRECTORY): - self.assertEquals(123, - self.device.FileSize('/data/local/tmp/foo.txt')) + self.assertEquals(123, self.device.FileSize('/data/local/tmp/foo.txt')) def testFileSize_doesNotExist(self): with self.assertCall( @@ -2425,7 +2746,6 @@ class DeviceUtilsFileSizeTest(DeviceUtilsTest): class DeviceUtilsSetJavaAssertsTest(DeviceUtilsTest): - def testSetJavaAsserts_enable(self): with self.assertCalls( (self.call.device.ReadFile(self.device.LOCAL_PROPERTIES_PATH), @@ -2475,14 +2795,14 @@ class DeviceUtilsSetJavaAssertsTest(DeviceUtilsTest): class DeviceUtilsEnsureCacheInitializedTest(DeviceUtilsTest): - def testEnsureCacheInitialized_noCache_success(self): self.assertIsNone(self.device._cache['token']) with self.assertCall( self.call.device.RunShellCommand( AnyStringWith('getprop'), - shell=True, check_return=True, large_output=True), - ['/sdcard', 'TOKEN']): + shell=True, + check_return=True, + large_output=True), ['/sdcard', 'TOKEN']): self.device._EnsureCacheInitialized() self.assertIsNotNone(self.device._cache['token']) @@ -2491,8 +2811,9 @@ class DeviceUtilsEnsureCacheInitializedTest(DeviceUtilsTest): with self.assertCall( self.call.device.RunShellCommand( AnyStringWith('getprop'), - shell=True, check_return=True, large_output=True), - self.TimeoutError()): + shell=True, + check_return=True, + large_output=True), self.TimeoutError()): with self.assertRaises(device_errors.CommandTimeoutError): self.device._EnsureCacheInitialized() self.assertIsNone(self.device._cache['token']) @@ -2505,24 +2826,23 @@ class DeviceUtilsEnsureCacheInitializedTest(DeviceUtilsTest): class DeviceUtilsGetPropTest(DeviceUtilsTest): - def testGetProp_exists(self): with self.assertCall( - self.call.device.RunShellCommand( - ['getprop', 'test.property'], check_return=True, single_line=True, - timeout=self.device._default_timeout, - retries=self.device._default_retries), + self.call.device.RunShellCommand(['getprop', 'test.property'], + check_return=True, + single_line=True, + timeout=self.device._default_timeout, + retries=self.device._default_retries), 'property_value'): - self.assertEqual('property_value', - self.device.GetProp('test.property')) + self.assertEqual('property_value', self.device.GetProp('test.property')) def testGetProp_doesNotExist(self): with self.assertCall( - self.call.device.RunShellCommand( - ['getprop', 'property.does.not.exist'], - check_return=True, single_line=True, - timeout=self.device._default_timeout, - retries=self.device._default_retries), + self.call.device.RunShellCommand(['getprop', 'property.does.not.exist'], + check_return=True, + single_line=True, + timeout=self.device._default_timeout, + retries=self.device._default_retries), ''): self.assertEqual('', self.device.GetProp('property.does.not.exist')) @@ -2536,7 +2856,6 @@ class DeviceUtilsGetPropTest(DeviceUtilsTest): class DeviceUtilsSetPropTest(DeviceUtilsTest): - def testSetProp(self): with self.assertCall( self.call.device.RunShellCommand( @@ -2576,26 +2895,28 @@ class DeviceUtilsListProcessesTest(DeviceUtilsTest): return [line for line in self.sample_output if substring in line] def testListProcesses_sdkGreaterThanNougatMR1(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=(version_codes.NOUGAT_MR1 + 1)): - with self.patch_call(self.call.device.build_id, - return_value='ZZZ99Z'): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=(version_codes.NOUGAT_MR1 + 1)): + with self.patch_call(self.call.device.build_id, return_value='ZZZ99Z'): with self.assertCall( self.call.device._RunPipedShellCommand( 'ps -e | grep -F example.process'), []): self.device.ListProcesses('example.process') def testListProcesses_noMatches(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F does.not.match'), self._grepOutput('does.not.match')): self.assertEqual([], self.device.ListProcesses('does.not.match')) def testListProcesses_oneMatch(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F one.match'), self._grepOutput('one.match')): @@ -2604,20 +2925,21 @@ class DeviceUtilsListProcessesTest(DeviceUtilsTest): self.device.ListProcesses('one.match')) def testListProcesses_multipleMatches(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F match'), self._grepOutput('match')): self.assertEqual( - Processes(('one.match', 1001, 100), - ('two.match', 1002, 100), + Processes(('one.match', 1001, 100), ('two.match', 1002, 100), ('three.match', 1003, 101)), self.device.ListProcesses('match')) def testListProcesses_quotable(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand("ps | grep -F 'my$process'"), self._grepOutput('my$process')): @@ -2627,34 +2949,36 @@ class DeviceUtilsListProcessesTest(DeviceUtilsTest): # Tests for the GetPids wrapper interface. def testGetPids_multipleInstances(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F foo'), self._grepOutput('foo')): - self.assertEqual( - {'foo': ['1236', '1578']}, - self.device.GetPids('foo')) + self.assertEqual({'foo': ['1236', '1578']}, self.device.GetPids('foo')) def testGetPids_allProcesses(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( - self.call.device.RunShellCommand( - ['ps'], check_return=True, large_output=True), + self.call.device.RunShellCommand(['ps'], + check_return=True, + large_output=True), self.sample_output): - self.assertEqual( - {'one.match': ['1001'], - 'two.match': ['1002'], - 'three.match': ['1003'], - 'my$process': ['1234'], - 'foo': ['1236', '1578']}, - self.device.GetPids()) + self.assertEqual({ + 'one.match': ['1001'], + 'two.match': ['1002'], + 'three.match': ['1003'], + 'my$process': ['1234'], + 'foo': ['1236', '1578'] + }, self.device.GetPids()) # Tests for the GetApplicationPids wrapper interface. def testGetApplicationPids_notFound(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F match'), self._grepOutput('match')): @@ -2662,47 +2986,48 @@ class DeviceUtilsListProcessesTest(DeviceUtilsTest): self.assertEqual([], self.device.GetApplicationPids('match')) def testGetApplicationPids_foundOne(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F one.match'), self._grepOutput('one.match')): self.assertEqual([1001], self.device.GetApplicationPids('one.match')) def testGetApplicationPids_foundMany(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F foo'), self._grepOutput('foo')): - self.assertEqual( - [1236, 1578], - self.device.GetApplicationPids('foo')) + self.assertEqual([1236, 1578], self.device.GetApplicationPids('foo')) def testGetApplicationPids_atMostOneNotFound(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F match'), self._grepOutput('match')): # No PIDs found, process name should be exact match. self.assertEqual( - None, - self.device.GetApplicationPids('match', at_most_one=True)) + None, self.device.GetApplicationPids('match', at_most_one=True)) def testGetApplicationPids_atMostOneFound(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F one.match'), self._grepOutput('one.match')): self.assertEqual( - 1001, - self.device.GetApplicationPids('one.match', at_most_one=True)) + 1001, self.device.GetApplicationPids('one.match', at_most_one=True)) def testGetApplicationPids_atMostOneFoundTooMany(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): with self.assertRaises(device_errors.CommandFailedError): with self.assertCall( self.call.device._RunPipedShellCommand('ps | grep -F foo'), @@ -2711,7 +3036,6 @@ class DeviceUtilsListProcessesTest(DeviceUtilsTest): class DeviceUtilsGetSetEnforce(DeviceUtilsTest): - def testGetEnforce_Enforcing(self): with self.assertCall(self.call.adb.Shell('getenforce'), 'Enforcing'): self.assertEqual(True, self.device.GetEnforce()) @@ -2725,44 +3049,37 @@ class DeviceUtilsGetSetEnforce(DeviceUtilsTest): self.assertEqual(None, self.device.GetEnforce()) def testSetEnforce_Enforcing(self): - with self.assertCalls( - (self.call.device.NeedsSU(), False), - (self.call.adb.Shell('setenforce 1'), '')): + with self.assertCalls((self.call.device.NeedsSU(), False), + (self.call.adb.Shell('setenforce 1'), '')): self.device.SetEnforce(enabled=True) def testSetEnforce_Permissive(self): - with self.assertCalls( - (self.call.device.NeedsSU(), False), - (self.call.adb.Shell('setenforce 0'), '')): + with self.assertCalls((self.call.device.NeedsSU(), False), + (self.call.adb.Shell('setenforce 0'), '')): self.device.SetEnforce(enabled=False) def testSetEnforce_EnforcingWithInt(self): - with self.assertCalls( - (self.call.device.NeedsSU(), False), - (self.call.adb.Shell('setenforce 1'), '')): + with self.assertCalls((self.call.device.NeedsSU(), False), + (self.call.adb.Shell('setenforce 1'), '')): self.device.SetEnforce(enabled=1) def testSetEnforce_PermissiveWithInt(self): - with self.assertCalls( - (self.call.device.NeedsSU(), False), - (self.call.adb.Shell('setenforce 0'), '')): + with self.assertCalls((self.call.device.NeedsSU(), False), + (self.call.adb.Shell('setenforce 0'), '')): self.device.SetEnforce(enabled=0) def testSetEnforce_EnforcingWithStr(self): - with self.assertCalls( - (self.call.device.NeedsSU(), False), - (self.call.adb.Shell('setenforce 1'), '')): + with self.assertCalls((self.call.device.NeedsSU(), False), + (self.call.adb.Shell('setenforce 1'), '')): self.device.SetEnforce(enabled='1') def testSetEnforce_PermissiveWithStr(self): - with self.assertCalls( - (self.call.device.NeedsSU(), False), - (self.call.adb.Shell('setenforce 0'), '')): + with self.assertCalls((self.call.device.NeedsSU(), False), + (self.call.adb.Shell('setenforce 0'), '')): self.device.SetEnforce(enabled='0') # Not recommended but it works! class DeviceUtilsGetWebViewUpdateServiceDumpTest(DeviceUtilsTest): - def testGetWebViewUpdateServiceDump_success(self): # Some of the lines of adb shell dumpsys webviewupdate: dumpsys_lines = [ @@ -2782,15 +3099,14 @@ class DeviceUtilsGetWebViewUpdateServiceDumpTest(DeviceUtilsTest): 'version too low'), ('com.chrome.canary is NOT installed.'), ] - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.OREO): + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.OREO): with self.assertCall( self.call.adb.Shell('dumpsys webviewupdate'), '\n'.join(dumpsys_lines)): update = self.device.GetWebViewUpdateServiceDump() self.assertTrue(update['FallbackLogicEnabled']) - self.assertEqual('com.android.chrome', - update['CurrentWebViewPackage']) + self.assertEqual('com.android.chrome', update['CurrentWebViewPackage']) self.assertEqual(12345, update['MinimumWebViewVersionCode']) # Order isn't really important, and we shouldn't have duplicates, so we # convert to sets. @@ -2799,49 +3115,49 @@ class DeviceUtilsGetWebViewUpdateServiceDumpTest(DeviceUtilsTest): 'com.google.android.apps.chrome', 'com.chrome.canary' } self.assertSetEqual(expected, set(update['WebViewPackages'].keys())) - self.assertEquals( - 'is installed/enabled for all users', - update['WebViewPackages']['com.android.chrome']) + self.assertEquals('is installed/enabled for all users', + update['WebViewPackages']['com.android.chrome']) self.assertEquals( 'is NOT installed/enabled for all users', update['WebViewPackages']['com.google.android.webview']) self.assertEquals( 'reason: SDK version too low', update['WebViewPackages']['com.google.android.apps.chrome']) - self.assertEquals( - 'is NOT installed.', - update['WebViewPackages']['com.chrome.canary']) + self.assertEquals('is NOT installed.', + update['WebViewPackages']['com.chrome.canary']) def testGetWebViewUpdateServiceDump_missingkey(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.OREO): - with self.assertCall(self.call.adb.Shell('dumpsys webviewupdate'), - 'Fallback logic enabled: true'): + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.OREO): + with self.assertCall( + self.call.adb.Shell('dumpsys webviewupdate'), + 'Fallback logic enabled: true'): with self.assertRaises(device_errors.CommandFailedError): self.device.GetWebViewUpdateServiceDump() def testGetWebViewUpdateServiceDump_noop(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.NOUGAT_MR1): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.NOUGAT_MR1): with self.assertCalls(): self.device.GetWebViewUpdateServiceDump() def testGetWebViewUpdateServiceDump_noPackage(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.OREO): - with self.assertCall(self.call.adb.Shell('dumpsys webviewupdate'), - 'Fallback logic enabled: true\n' - 'Current WebView package is null'): + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.OREO): + with self.assertCall( + self.call.adb.Shell('dumpsys webviewupdate'), + 'Fallback logic enabled: true\n' + 'Current WebView package is null'): update = self.device.GetWebViewUpdateServiceDump() self.assertEqual(True, update['FallbackLogicEnabled']) self.assertEqual(None, update['CurrentWebViewPackage']) class DeviceUtilsSetWebViewImplementationTest(DeviceUtilsTest): - def testSetWebViewImplementation_success(self): with self.patch_call( - self.call.device.GetApplicationPaths, return_value=['/any/path']): + self.call.device.IsApplicationInstalled, return_value=True): with self.assertCall( self.call.adb.Shell( 'cmd webviewupdate set-webview-implementation foo.org'), @@ -2849,7 +3165,8 @@ class DeviceUtilsSetWebViewImplementationTest(DeviceUtilsTest): self.device.SetWebViewImplementation('foo.org') def testSetWebViewImplementation_uninstalled(self): - with self.patch_call(self.call.device.GetApplicationPaths, return_value=[]): + with self.patch_call( + self.call.device.IsApplicationInstalled, return_value=False): with self.assertRaises(device_errors.CommandFailedError) as cfe: self.device.SetWebViewImplementation('foo.org') self.assertIn('is not installed', cfe.exception.message) @@ -2857,7 +3174,7 @@ class DeviceUtilsSetWebViewImplementationTest(DeviceUtilsTest): def _testSetWebViewImplementationHelper(self, mock_dump_sys, exception_message_substr): with self.patch_call( - self.call.device.GetApplicationPaths, return_value=['/any/path']): + self.call.device.IsApplicationInstalled, return_value=True): with self.assertCall( self.call.adb.Shell( 'cmd webviewupdate set-webview-implementation foo.org'), 'Oops!'): @@ -2894,11 +3211,28 @@ class DeviceUtilsSetWebViewImplementationTest(DeviceUtilsTest): self._testSetWebViewImplementationHelper(mock_dump_sys, 'WebView native library') - def testSetWebViewImplementation_lowTargetSdkVersion(self): - mock_dump_sys = {'WebViewPackages': {'foo.org': 'SDK version too low',}} - with self.patch_call(self.call.device.build_version_sdk, return_value=26): - self._testSetWebViewImplementationHelper(mock_dump_sys, - 'higher targetSdkVersion') + def testSetWebViewImplementation_lowTargetSdkVersion_finalizedSdk(self): + mock_dump_sys = {'WebViewPackages': {'foo.org': 'SDK version too low', }} + with self.assertCalls( + (self.call.device.GetApplicationTargetSdk('foo.org'), '29'), + (self.call.device.GetProp('ro.build.version.preview_sdk'), '0')): + with self.patch_call(self.call.device.build_version_sdk, return_value=30): + self._testSetWebViewImplementationHelper( + mock_dump_sys, + "has targetSdkVersion '29', but valid WebView providers must " + "target >= 30 on this device") + + def testSetWebViewImplementation_lowTargetSdkVersion_prefinalizedSdk(self): + mock_dump_sys = {'WebViewPackages': {'foo.org': 'SDK version too low', }} + with self.assertCalls( + (self.call.device.GetApplicationTargetSdk('foo.org'), '29'), + (self.call.device.GetProp('ro.build.version.preview_sdk'), '1'), + (self.call.device.GetProp('ro.build.version.codename'), 'R')): + with self.patch_call(self.call.device.build_version_sdk, return_value=29): + self._testSetWebViewImplementationHelper( + mock_dump_sys, + "targets a finalized SDK ('29'), but valid WebView providers must " + "target a pre-finalized SDK ('R') on this device") def testSetWebViewImplementation_lowVersionCode(self): mock_dump_sys = { @@ -2911,62 +3245,56 @@ class DeviceUtilsSetWebViewImplementationTest(DeviceUtilsTest): 'higher versionCode') def testSetWebViewImplementation_invalidSignature(self): - mock_dump_sys = { - 'WebViewPackages': { - 'foo.org': 'Incorrect signature', - } - } + mock_dump_sys = {'WebViewPackages': {'foo.org': 'Incorrect signature'}} self._testSetWebViewImplementationHelper(mock_dump_sys, 'signed with release keys') class DeviceUtilsSetWebViewFallbackLogicTest(DeviceUtilsTest): - def testSetWebViewFallbackLogic_False_success(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.NOUGAT): - with self.assertCall(self.call.adb.Shell( - 'cmd webviewupdate enable-redundant-packages'), 'Success'): + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.NOUGAT): + with self.assertCall( + self.call.adb.Shell('cmd webviewupdate enable-redundant-packages'), + 'Success'): self.device.SetWebViewFallbackLogic(False) def testSetWebViewFallbackLogic_True_success(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.NOUGAT): - with self.assertCall(self.call.adb.Shell( - 'cmd webviewupdate disable-redundant-packages'), 'Success'): + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.NOUGAT): + with self.assertCall( + self.call.adb.Shell('cmd webviewupdate disable-redundant-packages'), + 'Success'): self.device.SetWebViewFallbackLogic(True) def testSetWebViewFallbackLogic_failure(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.NOUGAT): - with self.assertCall(self.call.adb.Shell( - 'cmd webviewupdate enable-redundant-packages'), 'Oops!'): + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.NOUGAT): + with self.assertCall( + self.call.adb.Shell('cmd webviewupdate enable-redundant-packages'), + 'Oops!'): with self.assertRaises(device_errors.CommandFailedError): self.device.SetWebViewFallbackLogic(False) def testSetWebViewFallbackLogic_beforeNougat(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.MARSHMALLOW): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.MARSHMALLOW): with self.assertCalls(): self.device.SetWebViewFallbackLogic(False) def testSetWebViewFallbackLogic_afterPie(self): - # TODO(ntfschr): replace this with the Q constant when the SDK is public and - # the codename is finalized. - q_version_code = version_codes.PIE + 1 - with self.patch_call(self.call.device.build_version_sdk, - return_value=q_version_code): + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.Q): with self.assertCalls(): self.device.SetWebViewFallbackLogic(False) class DeviceUtilsTakeScreenshotTest(DeviceUtilsTest): - def testTakeScreenshot_fileNameProvided(self): with self.assertCalls( (mock.call.devil.android.device_temp_file.DeviceTempFile( - self.adb, suffix='.png'), - MockTempFile('/tmp/path/temp-123.png')), + self.adb, suffix='.png'), MockTempFile('/tmp/path/temp-123.png')), (self.call.adb.Shell('/system/bin/screencap -p /tmp/path/temp-123.png'), ''), self.call.device.PullFile('/tmp/path/temp-123.png', @@ -2975,7 +3303,6 @@ class DeviceUtilsTakeScreenshotTest(DeviceUtilsTest): class DeviceUtilsDismissCrashDialogIfNeededTest(DeviceUtilsTest): - def testDismissCrashDialogIfNeeded_crashedPageckageNotFound(self): sample_dumpsys_output = ''' WINDOW MANAGER WINDOWS (dumpsys window windows) @@ -2987,9 +3314,10 @@ WINDOW MANAGER WINDOWS (dumpsys window windows) mBaseLayer=211000 mSubLayer=0 mAnimLayer=211000+0=211000 mLastLayer=211000 ''' with self.assertCalls( - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), sample_dumpsys_output.split('\n'))): + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), + sample_dumpsys_output.split('\n'))): package_name = self.device.DismissCrashDialogIfNeeded() self.assertIsNone(package_name) @@ -3007,24 +3335,23 @@ WINDOW MANAGER WINDOWS (dumpsys window windows) mFocusedApp=AppWindowToken{470af6f token=Token{272ec24e ActivityRecord{t894}}} ''' with self.assertCalls( - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), sample_dumpsys_output.split('\n')), - (self.call.device.RunShellCommand( - ['input', 'keyevent', '22'], check_return=True)), - (self.call.device.RunShellCommand( - ['input', 'keyevent', '22'], check_return=True)), - (self.call.device.RunShellCommand( - ['input', 'keyevent', '66'], check_return=True)), - (self.call.device.RunShellCommand( - ['dumpsys', 'window', 'windows'], check_return=True, - large_output=True), [])): + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), + sample_dumpsys_output.split('\n')), (self.call.device.RunShellCommand( + ['input', 'keyevent', '22'], check_return=True)), + (self.call.device.RunShellCommand(['input', 'keyevent', '22'], + check_return=True)), + (self.call.device.RunShellCommand(['input', 'keyevent', '66'], + check_return=True)), + (self.call.device.RunShellCommand(['dumpsys', 'window', 'windows'], + check_return=True, + large_output=True), [])): package_name = self.device.DismissCrashDialogIfNeeded() self.assertEqual(package_name, 'com.android.chrome') class DeviceUtilsClientCache(DeviceUtilsTest): - def testClientCache_twoCaches(self): self.device._cache['test'] = 0 client_cache_one = self.device.GetClientCache('ClientOne') @@ -3051,32 +3378,28 @@ class DeviceUtilsClientCache(DeviceUtilsTest): class DeviceUtilsHealthyDevicesTest(mock_calls.TestCase): - - def testHealthyDevices_emptyBlacklist_defaultDeviceArg(self): + def testHealthyDevices_emptyDenylist_defaultDeviceArg(self): test_serials = ['0123456789abcdef', 'fedcba9876543210'] with self.assertCalls( (mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.Devices(), [_AdbWrapperMock(s) for s in test_serials]), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM)): - blacklist = mock.NonCallableMock(**{'Read.return_value': []}) - devices = device_utils.DeviceUtils.HealthyDevices(blacklist) + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM), + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM)): + denylist = mock.NonCallableMock(**{'Read.return_value': []}) + devices = device_utils.DeviceUtils.HealthyDevices(denylist) for serial, device in zip(test_serials, devices): self.assertTrue(isinstance(device, device_utils.DeviceUtils)) self.assertEquals(serial, device.adb.GetDeviceSerial()) - def testHealthyDevices_blacklist_defaultDeviceArg(self): + def testHealthyDevices_denylist_defaultDeviceArg(self): test_serials = ['0123456789abcdef', 'fedcba9876543210'] with self.assertCalls( (mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.Devices(), [_AdbWrapperMock(s) for s in test_serials]), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM)): - blacklist = mock.NonCallableMock( + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM)): + denylist = mock.NonCallableMock( **{'Read.return_value': ['fedcba9876543210']}) - devices = device_utils.DeviceUtils.HealthyDevices(blacklist) + devices = device_utils.DeviceUtils.HealthyDevices(denylist) self.assertEquals(1, len(devices)) self.assertTrue(isinstance(devices[0], device_utils.DeviceUtils)) self.assertEquals('0123456789abcdef', devices[0].adb.GetDeviceSerial()) @@ -3086,10 +3409,8 @@ class DeviceUtilsHealthyDevicesTest(mock_calls.TestCase): with self.assertCalls( (mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.Devices(), [_AdbWrapperMock(s) for s in test_serials]), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM), + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM), + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM), (mock.call.devil.android.device_errors.MultipleDevicesError(mock.ANY), _MockMultipleDevicesError())): with self.assertRaises(_MockMultipleDevicesError): @@ -3100,8 +3421,7 @@ class DeviceUtilsHealthyDevicesTest(mock_calls.TestCase): with self.assertCalls( (mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.Devices(), [_AdbWrapperMock(s) for s in test_serials]), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM)): + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM)): devices = device_utils.DeviceUtils.HealthyDevices(device_arg=None) self.assertEquals(1, len(devices)) @@ -3132,10 +3452,8 @@ class DeviceUtilsHealthyDevicesTest(mock_calls.TestCase): with self.assertCalls( (mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.Devices(), [_AdbWrapperMock(s) for s in test_serials]), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM)): + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM), + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM)): devices = device_utils.DeviceUtils.HealthyDevices(device_arg=()) self.assertEquals(2, len(devices)) @@ -3169,8 +3487,10 @@ class DeviceUtilsHealthyDevicesTest(mock_calls.TestCase): with self.assertRaises(device_errors.NoDevicesError): device_utils.DeviceUtils.HealthyDevices(device_arg=[], retries=4) self.assertEquals(mock_restart.call_count, 4) - self.assertEquals(mock_sleep.call_args_list, [ - mock.call(2), mock.call(4), mock.call(8), mock.call(16)]) + self.assertEquals( + mock_sleep.call_args_list, + [mock.call(2), mock.call(4), + mock.call(8), mock.call(16)]) @mock.patch('time.sleep') @mock.patch('devil.android.device_utils.RestartServer') @@ -3187,14 +3507,16 @@ class DeviceUtilsHealthyDevicesTest(mock_calls.TestCase): (mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.Devices(), []), (mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.Devices(), [])): with self.assertRaises(device_errors.NoDevicesError): - with mock.patch.object( - mock_reset_import, 'reset_all_android_devices') as mock_reset: - device_utils.DeviceUtils.HealthyDevices(device_arg=[], retries=4, - enable_usb_resets=True) + with mock.patch.object(mock_reset_import, + 'reset_all_android_devices') as mock_reset: + device_utils.DeviceUtils.HealthyDevices( + device_arg=[], retries=4, enable_usb_resets=True) self.assertEquals(mock_reset.call_count, 1) self.assertEquals(mock_restart.call_count, 4) - self.assertEquals(mock_sleep.call_args_list, [ - mock.call(2), mock.call(4), mock.call(8), mock.call(16)]) + self.assertEquals( + mock_sleep.call_args_list, + [mock.call(2), mock.call(4), + mock.call(8), mock.call(16)]) def testHealthyDevices_ListDeviceArg(self): device_arg = ['0123456789abcdef', 'fedcba9876543210'] @@ -3211,13 +3533,11 @@ class DeviceUtilsHealthyDevicesTest(mock_calls.TestCase): with self.assertCalls( (mock.call.devil.android.sdk.adb_wrapper.AdbWrapper.Devices(), [_AdbWrapperMock(s) for s in test_serials]), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM)): + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM), + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM)): with self.assertRaises(device_errors.NoDevicesError): - device_utils.DeviceUtils.HealthyDevices(device_arg=[], retries=0, - abis=[abis.ARM_64]) + device_utils.DeviceUtils.HealthyDevices( + device_arg=[], retries=0, abis=[abis.ARM_64]) def testHealthyDevices_abisArg_filter_on_abi(self): test_serials = ['0123456789abcdef', 'fedcba9876543210'] @@ -3226,16 +3546,13 @@ class DeviceUtilsHealthyDevicesTest(mock_calls.TestCase): [_AdbWrapperMock(s) for s in test_serials]), (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM_64), - (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), - abis.ARM)): - devices = device_utils.DeviceUtils.HealthyDevices(device_arg=[], - retries=0, - abis=[abis.ARM_64]) + (mock.call.devil.android.device_utils.DeviceUtils.GetABI(), abis.ARM)): + devices = device_utils.DeviceUtils.HealthyDevices( + device_arg=[], retries=0, abis=[abis.ARM_64]) self.assertEquals(1, len(devices)) class DeviceUtilsRestartAdbdTest(DeviceUtilsTest): - def testAdbdRestart(self): mock_temp_file = '/sdcard/temp-123.sh' with self.assertCalls( @@ -3257,60 +3574,62 @@ class DeviceUtilsGrantPermissionsTest(DeviceUtilsTest): output, status = result + '\n', 1 else: output, status = '', 0 - results.append( - '{output}{sep}{permission}{sep}{status}{sep}\n'.format( - output=output, - permission=permission, - status=status, - sep=device_utils._SHELL_OUTPUT_SEPARATOR - )) - return ( - self.call.device.RunShellCommand( - AnyStringWith(fragment), - shell=True, raw_output=True, large_output=True, check_return=True), - ''.join(results)) + results.append('{output}{sep}{permission}{sep}{status}{sep}\n'.format( + output=output, + permission=permission, + status=status, + sep=device_utils._SHELL_OUTPUT_SEPARATOR)) + return (self.call.device.RunShellCommand( + AnyStringWith(fragment), + shell=True, + raw_output=True, + large_output=True, + check_return=True), ''.join(results)) def testGrantPermissions_none(self): self.device.GrantPermissions('package', []) - def testGrantPermissions_underM(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): - self.device.GrantPermissions('package', ['p1']) - def testGrantPermissions_one(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.MARSHMALLOW): - with self.assertCalls( - self._PmGrantShellCall('package', {'p1': 0})): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.MARSHMALLOW): + with self.assertCalls(self._PmGrantShellCall('package', {'p1': 0})): self.device.GrantPermissions('package', ['p1']) def testGrantPermissions_multiple(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.MARSHMALLOW): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.MARSHMALLOW): with self.assertCalls( - self._PmGrantShellCall('package', {'p1': 0, 'p2': 0})): + self._PmGrantShellCall('package', { + 'p1': 0, + 'p2': 0 + })): self.device.GrantPermissions('package', ['p1', 'p2']) def testGrantPermissions_WriteExtrnalStorage(self): WRITE = 'android.permission.WRITE_EXTERNAL_STORAGE' READ = 'android.permission.READ_EXTERNAL_STORAGE' with PatchLogger() as logger: - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.MARSHMALLOW): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.MARSHMALLOW): with self.assertCalls( - self._PmGrantShellCall('package', {READ: 0, WRITE: 0})): + self._PmGrantShellCall('package', { + READ: 0, + WRITE: 0 + })): self.device.GrantPermissions('package', [WRITE]) self.assertEqual(logger.warnings, []) - def testGrantPermissions_BlackList(self): + def testGrantPermissions_DenyList(self): with PatchLogger() as logger: - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.MARSHMALLOW): - with self.assertCalls( - self._PmGrantShellCall('package', {'p1': 0})): - self.device.GrantPermissions( - 'package', ['p1', 'foo.permission.C2D_MESSAGE']) + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.MARSHMALLOW): + with self.assertCalls(self._PmGrantShellCall('package', {'p1': 0})): + self.device.GrantPermissions('package', + ['p1', 'foo.permission.C2D_MESSAGE']) self.assertEqual(logger.warnings, []) def testGrantPermissions_unchangeablePermision(self): @@ -3318,13 +3637,14 @@ class DeviceUtilsGrantPermissionsTest(DeviceUtilsTest): 'Operation not allowed: java.lang.SecurityException: ' 'Permission UNCHANGEABLE is not a changeable permission type') with PatchLogger() as logger: - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.MARSHMALLOW): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.MARSHMALLOW): with self.assertCalls( self._PmGrantShellCall('package', {'UNCHANGEABLE': error_message})): self.device.GrantPermissions('package', ['UNCHANGEABLE']) - self.assertEqual( - logger.warnings, [mock.ANY, AnyStringWith('UNCHANGEABLE')]) + self.assertEqual(logger.warnings, + [mock.ANY, AnyStringWith('UNCHANGEABLE')]) class DeviecUtilsIsScreenOn(DeviceUtilsTest): @@ -3335,53 +3655,49 @@ class DeviecUtilsIsScreenOn(DeviceUtilsTest): _K_SCREEN_OFF = ['mScreenOn=false'] def testIsScreenOn_onPreL(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.KITKAT): - with self.assertCalls( - (self.call.device._RunPipedShellCommand( - 'dumpsys input_method | grep mScreenOn'), self._K_SCREEN_ON)): + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.KITKAT): + with self.assertCalls((self.call.device._RunPipedShellCommand( + 'dumpsys input_method | grep mScreenOn'), self._K_SCREEN_ON)): self.assertTrue(self.device.IsScreenOn()) def testIsScreenOn_onL(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): - with self.assertCalls( - (self.call.device._RunPipedShellCommand( - 'dumpsys input_method | grep mInteractive'), self._L_SCREEN_ON)): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): + with self.assertCalls((self.call.device._RunPipedShellCommand( + 'dumpsys input_method | grep mInteractive'), self._L_SCREEN_ON)): self.assertTrue(self.device.IsScreenOn()) def testIsScreenOn_offPreL(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.KITKAT): - with self.assertCalls( - (self.call.device._RunPipedShellCommand( - 'dumpsys input_method | grep mScreenOn'), self._K_SCREEN_OFF)): + with self.patch_call( + self.call.device.build_version_sdk, return_value=version_codes.KITKAT): + with self.assertCalls((self.call.device._RunPipedShellCommand( + 'dumpsys input_method | grep mScreenOn'), self._K_SCREEN_OFF)): self.assertFalse(self.device.IsScreenOn()) def testIsScreenOn_offL(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): - with self.assertCalls( - (self.call.device._RunPipedShellCommand( - 'dumpsys input_method | grep mInteractive'), self._L_SCREEN_OFF)): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): + with self.assertCalls((self.call.device._RunPipedShellCommand( + 'dumpsys input_method | grep mInteractive'), self._L_SCREEN_OFF)): self.assertFalse(self.device.IsScreenOn()) def testIsScreenOn_noOutput(self): - with self.patch_call(self.call.device.build_version_sdk, - return_value=version_codes.LOLLIPOP): - with self.assertCalls( - (self.call.device._RunPipedShellCommand( - 'dumpsys input_method | grep mInteractive'), [])): + with self.patch_call( + self.call.device.build_version_sdk, + return_value=version_codes.LOLLIPOP): + with self.assertCalls((self.call.device._RunPipedShellCommand( + 'dumpsys input_method | grep mInteractive'), [])): with self.assertRaises(device_errors.CommandFailedError): self.device.IsScreenOn() class DeviecUtilsSetScreen(DeviceUtilsTest): - @mock.patch('time.sleep', mock.Mock()) def testSetScren_alreadySet(self): - with self.assertCalls( - (self.call.device.IsScreenOn(), False)): + with self.assertCalls((self.call.device.IsScreenOn(), False)): self.device.SetScreen(False) @mock.patch('time.sleep', mock.Mock()) @@ -3410,38 +3726,35 @@ class DeviecUtilsSetScreen(DeviceUtilsTest): (self.call.device.IsScreenOn(), False)): self.device.SetScreen(False) + class DeviecUtilsLoadCacheData(DeviceUtilsTest): + def testInvalidJson(self): + self.assertFalse(self.device.LoadCacheData('')) def testTokenMissing(self): - with self.assertCalls( - self.EnsureCacheInitialized()): + with self.assertCalls(self.EnsureCacheInitialized()): self.assertFalse(self.device.LoadCacheData('{}')) def testTokenStale(self): - with self.assertCalls( - self.EnsureCacheInitialized()): + with self.assertCalls(self.EnsureCacheInitialized()): self.assertFalse(self.device.LoadCacheData('{"token":"foo"}')) def testTokenMatches(self): - with self.assertCalls( - self.EnsureCacheInitialized()): + with self.assertCalls(self.EnsureCacheInitialized()): self.assertTrue(self.device.LoadCacheData('{"token":"TOKEN"}')) def testDumpThenLoad(self): - with self.assertCalls( - self.EnsureCacheInitialized()): + with self.assertCalls(self.EnsureCacheInitialized()): data = json.loads(self.device.DumpCacheData()) data['token'] = 'TOKEN' self.assertTrue(self.device.LoadCacheData(json.dumps(data))) class DeviceUtilsGetIMEITest(DeviceUtilsTest): - def testSuccessfulDumpsys(self): - dumpsys_output = ( - 'Phone Subscriber Info:' - ' Phone Type = GSM' - ' Device ID = 123454321') + dumpsys_output = ('Phone Subscriber Info:' + ' Phone Type = GSM' + ' Device ID = 123454321') with self.assertCalls( (self.call.device.GetProp('ro.build.version.sdk', cache=True), '19'), (self.call.adb.Shell('dumpsys iphonesubinfo'), dumpsys_output)): @@ -3476,7 +3789,6 @@ class DeviceUtilsGetIMEITest(DeviceUtilsTest): class DeviceUtilsChangeOwner(DeviceUtilsTest): - def testChangeOwner(self): with self.assertCalls( (self.call.device.RunShellCommand( @@ -3486,18 +3798,16 @@ class DeviceUtilsChangeOwner(DeviceUtilsTest): class DeviceUtilsChangeSecurityContext(DeviceUtilsTest): - def testChangeSecurityContext(self): - with self.assertCalls( - (self.call.device.RunShellCommand( - ['chcon', 'u:object_r:system_data_file:s0', '/path', '/path2'], - as_root=device_utils._FORCE_SU, check_return=True))): + with self.assertCalls((self.call.device.RunShellCommand( + ['chcon', 'u:object_r:system_data_file:s0', '/path', '/path2'], + as_root=device_utils._FORCE_SU, + check_return=True))): self.device.ChangeSecurityContext('u:object_r:system_data_file:s0', ['/path', '/path2']) class DeviceUtilsLocale(DeviceUtilsTest): - def testLocaleLegacy(self): with self.assertCalls( (self.call.device.GetProp('persist.sys.locale', cache=False), ''), @@ -3514,11 +3824,10 @@ class DeviceUtilsLocale(DeviceUtilsTest): self.assertEquals(self.device.GetLocale(), ('en', 'US-sw')) def testBadLocale(self): - with self.assertCalls( - (self.call.device.GetProp('persist.sys.locale', cache=False), 'en')): + with self.assertCalls((self.call.device.GetProp( + 'persist.sys.locale', cache=False), 'en')): self.assertEquals(self.device.GetLocale(), ('', '')) - def testLanguageAndCountryLegacy(self): with self.assertCalls( (self.call.device.GetProp('persist.sys.locale', cache=False), ''), @@ -3538,6 +3847,138 @@ class DeviceUtilsLocale(DeviceUtilsTest): self.assertEquals(self.device.GetCountry(), 'US') +class IterPushableComponentsTest(unittest.TestCase): + @classmethod + @contextlib.contextmanager + def sampleLayout(cls): + Layout = collections.namedtuple('Layout', [ + 'root', 'basic_file', 'symlink_file', 'symlink_dir', + 'dir_with_symlinks', 'dir_without_symlinks' + ]) + + with tempfile_ext.NamedTemporaryDirectory() as layout_root: + dir1 = os.path.join(layout_root, 'dir1') + os.makedirs(dir1) + + basic_file = os.path.join(dir1, 'file1.txt') + with open(basic_file, 'w') as f: + f.write('hello world') + + symlink = os.path.join(dir1, 'symlink.txt') + os.symlink(basic_file, symlink) + + dir2 = os.path.join(layout_root, 'dir2') + os.makedirs(dir2) + + with open(os.path.join(dir2, 'file2.txt'), 'w') as f: + f.write('goodnight moon') + + symlink_dir = os.path.join(layout_root, 'dir3') + os.symlink(dir2, symlink_dir) + + yield Layout(layout_root, basic_file, symlink, symlink_dir, dir1, dir2) + + def testFile(self): + with self.sampleLayout() as layout: + device_path = '/sdcard/basic_file' + + expected = [(layout.basic_file, device_path, True)] + actual = list( + device_utils._IterPushableComponents(layout.basic_file, device_path)) + self.assertItemsEqual(expected, actual) + + def testSymlinkFile(self): + with self.sampleLayout() as layout: + device_path = '/sdcard/basic_symlink' + + expected = [(os.path.realpath(layout.symlink_file), device_path, False)] + actual = list( + device_utils._IterPushableComponents(layout.symlink_file, + device_path)) + self.assertItemsEqual(expected, actual) + + def testDirectoryWithNoSymlink(self): + with self.sampleLayout() as layout: + device_path = '/sdcard/basic_directory' + + expected = [(layout.dir_without_symlinks, device_path, True)] + actual = list( + device_utils._IterPushableComponents(layout.dir_without_symlinks, + device_path)) + self.assertItemsEqual(expected, actual) + + def testDirectoryWithSymlink(self): + with self.sampleLayout() as layout: + device_path = '/sdcard/directory' + + expected = [ + (layout.basic_file, + posixpath.join(device_path, os.path.basename(layout.basic_file)), + True), + (os.path.realpath(layout.symlink_file), + posixpath.join(device_path, os.path.basename(layout.symlink_file)), + False), + ] + actual = list( + device_utils._IterPushableComponents(layout.dir_with_symlinks, + device_path)) + self.assertItemsEqual(expected, actual) + + def testSymlinkDirectory(self): + with self.sampleLayout() as layout: + device_path = '/sdcard/directory' + + expected = [(os.path.realpath(layout.symlink_dir), device_path, False)] + actual = list( + device_utils._IterPushableComponents(layout.symlink_dir, device_path)) + self.assertItemsEqual(expected, actual) + + def testDirectoryWithNestedSymlink(self): + with self.sampleLayout() as layout: + device_path = '/sdcard/directory' + + expected = [ + (layout.dir_without_symlinks, + posixpath.join(device_path, + os.path.basename(layout.dir_without_symlinks)), True), + (layout.basic_file, + posixpath.join( + device_path, + *os.path.split(os.path.relpath(layout.basic_file, layout.root))), + True), + (os.path.realpath(layout.symlink_file), + posixpath.join( + device_path, + *os.path.split( + os.path.relpath(layout.symlink_file, layout.root))), False), + (os.path.realpath(layout.symlink_dir), + posixpath.join( + device_path, + *os.path.split(os.path.relpath(layout.symlink_dir, + layout.root))), False), + ] + actual = list( + device_utils._IterPushableComponents(layout.root, device_path)) + self.assertItemsEqual(expected, actual) + + +class DeviceUtilsGetTracingPathTest(DeviceUtilsTest): + def testGetTracingPath_hasDebugfs(self): + with self.assertCalls( + (self.call.device.RunShellCommand(['mount'], retries=0, + timeout=10, check_return=True), + ['debugfs on /sys/kernel/debug', 'proc on /proc'])): + self.assertEquals('/sys/kernel/debug/tracing', + self.device.GetTracingPath()) + + def testGetTracingPath_noDebugfs(self): + with self.assertCalls( + (self.call.device.RunShellCommand(['mount'], retries=0, + timeout=10, check_return=True), + ['proc on /proc'])): + self.assertEquals('/sys/kernel/tracing', self.device.GetTracingPath()) + + if __name__ == '__main__': logging.getLogger().setLevel(logging.DEBUG) unittest.main(verbosity=2) diff --git a/catapult/devil/devil/android/fastboot_utils.py b/catapult/devil/devil/android/fastboot_utils.py index 3621d7fb..d8ca7d20 100644 --- a/catapult/devil/devil/android/fastboot_utils.py +++ b/catapult/devil/devil/android/fastboot_utils.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Provides a variety of device interactions based on fastboot.""" # pylint: disable=unused-argument @@ -14,6 +13,7 @@ import re from devil.android import decorators from devil.android import device_errors +from devil.android import device_utils from devil.android.sdk import fastboot from devil.utils import timeout_retry @@ -23,58 +23,62 @@ _DEFAULT_TIMEOUT = 30 _DEFAULT_RETRIES = 3 _FASTBOOT_REBOOT_TIMEOUT = 10 * _DEFAULT_TIMEOUT _KNOWN_PARTITIONS = collections.OrderedDict([ - ('bootloader', {'image': 'bootloader*.img', 'restart': True}), - ('radio', {'image': 'radio*.img', 'restart': True}), - ('boot', {'image': 'boot.img'}), - ('recovery', {'image': 'recovery.img'}), - ('system', {'image': 'system.img'}), - ('userdata', {'image': 'userdata.img', 'wipe_only': True}), - ('cache', {'image': 'cache.img', 'wipe_only': True}), - ('vendor', {'image': 'vendor*.img', 'optional': True}), - ]) + ('bootloader', { + 'image': 'bootloader*.img', + 'restart': True + }), + ('radio', { + 'image': 'radio*.img', + 'restart': True + }), + ('boot', { + 'image': 'boot.img' + }), + # recovery.img moved into boot.img for A/B devices. See: + # https://source.android.com/devices/tech/ota/ab/ab_implement#recovery + ('recovery', { + 'image': 'recovery.img', + 'optional': lambda fu: fu.supports_ab + }), + ('system', { + 'image': 'system.img' + }), + ('userdata', { + 'image': 'userdata.img', + 'wipe_only': True + }), + # cache.img deprecated for A/B devices. See: + # https://source.android.com/devices/tech/ota/ab/ab_implement#cache + ('cache', { + 'image': 'cache.img', + 'wipe_only': True, + 'optional': lambda fu: fu.supports_ab + }), + ('vendor', { + 'image': 'vendor*.img', + 'optional': lambda _: True + }), + ('dtbo', { + 'image': 'dtbo.img', + 'optional': lambda fu: not fu.requires_dtbo + }), + ('vbmeta', { + 'image': 'vbmeta.img', + 'optional': lambda fu: not fu.requires_vbmeta + }), +]) ALL_PARTITIONS = _KNOWN_PARTITIONS.keys() -def _FindAndVerifyPartitionsAndImages(partitions, directory): - """Validate partitions and images. - - Validate all partition names and partition directories. Cannot stop mid - flash so its important to validate everything first. - - Args: - Partitions: partitions to be tested. - directory: directory containing the images. - - Returns: - Dictionary with exact partition, image name mapping. - """ - - files = os.listdir(directory) - return_dict = collections.OrderedDict() - - def find_file(pattern): - for filename in files: - if fnmatch.fnmatch(filename, pattern): - return os.path.join(directory, filename) - return None - for partition in partitions: - partition_info = _KNOWN_PARTITIONS[partition] - image_file = find_file(partition_info['image']) - if image_file: - return_dict[partition] = image_file - elif not partition_info.get('optional'): - raise device_errors.FastbootCommandFailedError( - 'Failed to flash device. Could not find image for %s.', - partition_info['image']) - return return_dict - - class FastbootUtils(object): _FASTBOOT_WAIT_TIME = 1 _BOARD_VERIFICATION_FILE = 'android-info.txt' - def __init__(self, device, fastbooter=None, default_timeout=_DEFAULT_TIMEOUT, + def __init__(self, + device=None, + fastbooter=None, + default_timeout=_DEFAULT_TIMEOUT, default_retries=_DEFAULT_RETRIES): """FastbootUtils constructor. @@ -83,23 +87,99 @@ class FastbootUtils(object): fastboot.FlashDevice('/path/to/build/directory') Args: - device: A DeviceUtils instance. - fastbooter: Optional fastboot object. If none is passed, one will - be created. + device: A DeviceUtils instance. Optional if a Fastboot instance was + passed. + fastbooter: A fastboot.Fastboot instance. Optional if a DeviceUtils + instance was passed. default_timeout: An integer containing the default number of seconds to wait for an operation to complete if no explicit value is provided. default_retries: An integer containing the default number or times an operation should be retried on failure if no explicit value is provided. """ - self._device = device - self._board = device.product_board - self._serial = str(device) - self._default_timeout = default_timeout - self._default_retries = default_retries + if not device and not fastbooter: + raise ValueError("One of 'device' or 'fastbooter' must be passed.") + + if device: + self._device = device + self._serial = str(device) + self._board = device.product_board + if not fastbooter: + self.fastboot = fastboot.Fastboot(self._serial) + if fastbooter: + self._serial = str(fastbooter) self.fastboot = fastbooter - else: - self.fastboot = fastboot.Fastboot(self._serial) + self._board = fastbooter.GetVar('product') + if not device: + self._device = device_utils.DeviceUtils(self._serial) + + self._default_timeout = default_timeout + self._default_retries = default_retries + + self._supports_ab = None + self._requires_dtbo = None + self._requires_vbmeta = None + + @property + def supports_ab(self): + """returns boolean to indicate if a device supports A/B updates. + + It appears that boards which support A/B updates have different partition + requirements when flashing. + """ + if self._supports_ab is None: + if self.IsFastbootMode(): + try: + # According to https://bit.ly/2XIuICQ, slot-count is used to + # determine if a device supports A/B updates. + slot_count = self.fastboot.GetVar('slot-count') or '0' + self._supports_ab = int(slot_count) >= 2 + except device_errors.FastbootCommandFailedError: + self._supports_ab = False + else: + # According to https://bit.ly/2UlJkGa and https://bit.ly/2MG8CL0, + # the property 'ro.build.ab_update' will be defined if the device + # supports A/B system updates. + self._supports_ab = self._device.GetProp('ro.build.ab_update') == 'true' + + return self._supports_ab + + @property + def requires_dtbo(self): + if self._requires_dtbo is None: + if self.IsFastbootMode(): + try: + self._requires_dtbo = self.fastboot.GetVar('has-slot:dtbo') == 'yes' + except device_errors.FastbootCommandFailedError: + self._requires_dtbo = False + else: + # This prop will be set when a device supports dtbo. + # See https://bit.ly/2VUjBp0. + # Checking if this prop has a non-empty value should be good enough. + self._requires_dtbo = len(self._device.GetProp('ro.boot.dtbo_idx')) > 0 + + return self._requires_dtbo + + @property + def requires_vbmeta(self): + if self._requires_vbmeta is None: + if self.IsFastbootMode(): + try: + self._requires_vbmeta = self.fastboot.GetVar( + 'has-slot:vbmeta') == 'yes' + except device_errors.FastbootCommandFailedError: + self._requires_vbmeta = False + else: + # This prop will be set when a device uses Android Verified Boot (avb). + # See https://bit.ly/2CbsO5z. + # Checking if this prop has a non-empty value should be good enough. + self._requires_vbmeta = len( + self._device.GetProp('ro.boot.vbmeta.digest')) > 0 + + return self._requires_vbmeta + + def IsFastbootMode(self): + return self._serial in (str(d) for d in self.fastboot.Devices()) @decorators.WithTimeoutAndRetriesFromInstance() def WaitForFastbootMode(self, timeout=None, retries=None): @@ -107,10 +187,8 @@ class FastbootUtils(object): This waits for the device serial to show up in fastboot devices output. """ - def fastboot_mode(): - return any(self._serial == str(d) for d in self.fastboot.Devices()) - - timeout_retry.WaitFor(fastboot_mode, wait_period=self._FASTBOOT_WAIT_TIME) + timeout_retry.WaitFor(self.IsFastbootMode, + wait_period=self._FASTBOOT_WAIT_TIME) @decorators.WithTimeoutAndRetriesFromInstance( min_default_timeout=_FASTBOOT_REBOOT_TIMEOUT) @@ -119,14 +197,19 @@ class FastbootUtils(object): Roots phone if needed, then reboots phone into fastboot mode and waits. """ + if self.IsFastbootMode(): + return self._device.EnableRoot() self._device.adb.Reboot(to_bootloader=True) self.WaitForFastbootMode() @decorators.WithTimeoutAndRetriesFromInstance( min_default_timeout=_FASTBOOT_REBOOT_TIMEOUT) - def Reboot( - self, bootloader=False, wait_for_reboot=True, timeout=None, retries=None): + def Reboot(self, + bootloader=False, + wait_for_reboot=True, + timeout=None, + retries=None): """Reboots out of fastboot mode. It reboots the phone either back into fastboot, or to a regular boot. It @@ -153,20 +236,16 @@ class FastbootUtils(object): directory: directory where build files are located. """ files = os.listdir(directory) - board_regex = re.compile(r'require board=(\w+)') + board_regex = re.compile(r'require board=([\w|]+)') if self._BOARD_VERIFICATION_FILE in files: with open(os.path.join(directory, self._BOARD_VERIFICATION_FILE)) as f: for line in f: m = board_regex.match(line) - if m: - board_name = m.group(1) - if board_name == self._board: - return True - elif board_name: - return False - else: - logger.warning('No board type found in %s.', - self._BOARD_VERIFICATION_FILE) + if m and m.group(1): + return self._board in m.group(1).split('|') + else: + logger.warning('No board type found in %s.', + self._BOARD_VERIFICATION_FILE) else: logger.warning('%s not found. Unable to use it to verify device.', self._BOARD_VERIFICATION_FILE) @@ -203,19 +282,58 @@ class FastbootUtils(object): 'device type. Run again with force=True to force flashing with an ' 'unverified board.') - flash_image_files = _FindAndVerifyPartitionsAndImages(partitions, directory) + flash_image_files = self._FindAndVerifyPartitionsAndImages( + partitions, directory) partitions = flash_image_files.keys() for partition in partitions: if _KNOWN_PARTITIONS[partition].get('wipe_only') and not wipe: - logger.info( - 'Not flashing in wipe mode. Skipping partition %s.', partition) + logger.info('Not flashing in wipe mode. Skipping partition %s.', + partition) else: - logger.info( - 'Flashing %s with %s', partition, flash_image_files[partition]) + logger.info('Flashing %s with %s', partition, + flash_image_files[partition]) self.fastboot.Flash(partition, flash_image_files[partition]) if _KNOWN_PARTITIONS[partition].get('restart', False): self.Reboot(bootloader=True) + def _FindAndVerifyPartitionsAndImages(self, partitions, directory): + """Validate partitions and images. + + Validate all partition names and partition directories. Cannot stop mid + flash so its important to validate everything first. + + Args: + Partitions: partitions to be tested. + directory: directory containing the images. + + Returns: + Dictionary with exact partition, image name mapping. + """ + + files = os.listdir(directory) + return_dict = collections.OrderedDict() + + def find_file(pattern): + for filename in files: + if fnmatch.fnmatch(filename, pattern): + return os.path.join(directory, filename) + return None + + for partition in partitions: + partition_info = _KNOWN_PARTITIONS[partition] + image_file = find_file(partition_info['image']) + if image_file: + return_dict[partition] = image_file + elif (not 'optional' in partition_info + or not partition_info['optional'](self)): + raise device_errors.FastbootCommandFailedError( + [], + '', + message='Failed to flash device%s. Could not find image for %s.' % + (' which supports A/B updates' if self.supports_ab else '', + partition_info['image'])) + return return_dict + @contextlib.contextmanager def FastbootMode(self, wait_for_reboot=True, timeout=None, retries=None): """Context manager that enables fastboot mode, and reboots after. @@ -227,11 +345,12 @@ class FastbootUtils(object): """ self.EnableFastbootMode() self.fastboot.SetOemOffModeCharge(False) - try: - yield self - finally: - self.fastboot.SetOemOffModeCharge(True) - self.Reboot(wait_for_reboot=wait_for_reboot) + yield self + # If something went wrong while it was in fastboot mode (eg: a failed + # flash) rebooting may be harmful or cause boot loops. So only reboot if + # no exception was thrown. + self.fastboot.SetOemOffModeCharge(True) + self.Reboot(wait_for_reboot=wait_for_reboot) def FlashDevice(self, directory, partitions=None, wipe=False): """Flash device with build in |directory|. @@ -241,7 +360,6 @@ class FastbootUtils(object): use with care. Args: - fastboot: A FastbootUtils instance. directory: Directory with build files. wipe: Wipes cache and userdata if set to true. partitions: List of partitions to flash. Defaults to all. diff --git a/catapult/devil/devil/android/fastboot_utils_test.py b/catapult/devil/devil/android/fastboot_utils_test.py index 05629746..1ad73190 100755 --- a/catapult/devil/devil/android/fastboot_utils_test.py +++ b/catapult/devil/devil/android/fastboot_utils_test.py @@ -2,7 +2,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Unit tests for the contents of fastboot_utils.py """ @@ -27,7 +26,8 @@ with devil_env.SysPath(devil_env.PYMOCK_PATH): _BOARD = 'board_type' _SERIAL = '0123456789abcdef' _PARTITIONS = [ - 'bootloader', 'radio', 'boot', 'recovery', 'system', 'userdata', 'cache'] + 'bootloader', 'radio', 'boot', 'recovery', 'system', 'userdata', 'cache' +] _IMAGES = collections.OrderedDict([ ('bootloader', 'bootloader.img'), ('radio', 'radio.img'), @@ -35,14 +35,13 @@ _IMAGES = collections.OrderedDict([ ('recovery', 'recovery.img'), ('system', 'system.img'), ('userdata', 'userdata.img'), - ('cache', 'cache.img') + ('cache', 'cache.img'), ]) _VALID_FILES = [_BOARD + '.zip', 'android-info.txt'] _INVALID_FILES = ['test.zip', 'android-info.txt'] class MockFile(object): - def __init__(self, name='/tmp/some/file'): self.file = mock.MagicMock(spec=file) self.file.name = name @@ -74,32 +73,35 @@ def _DeviceUtilsMock(test_serial): class FastbootUtilsTest(mock_calls.TestCase): - def setUp(self): self.device_utils_mock = _DeviceUtilsMock(_SERIAL) self.fastboot_wrapper = _FastbootWrapperMock(_SERIAL) self.fastboot = fastboot_utils.FastbootUtils( - self.device_utils_mock, fastbooter=self.fastboot_wrapper, - default_timeout=2, default_retries=0) + device=self.device_utils_mock, + fastbooter=self.fastboot_wrapper, + default_timeout=2, + default_retries=0) self.fastboot._board = _BOARD + def FastbootCommandFailedError(self, + args=None, + output=None, + status=None, + msg=None): + return mock.Mock(side_effect=device_errors.FastbootCommandFailedError( + args, output, status, msg, str(self.device_utils_mock))) -class FastbootUtilsInitTest(FastbootUtilsTest): +class FastbootUtilsInitTest(FastbootUtilsTest): def testInitWithDeviceUtil(self): f = fastboot_utils.FastbootUtils(self.device_utils_mock) self.assertEqual(str(self.device_utils_mock), str(f._device)) def testInitWithMissing_fails(self): + with self.assertRaises(ValueError): + fastboot_utils.FastbootUtils(device=None, fastbooter=None) with self.assertRaises(AttributeError): - fastboot_utils.FastbootUtils(None) - with self.assertRaises(AttributeError): - fastboot_utils.FastbootUtils('') - - def testPartitionOrdering(self): - parts = ['bootloader', 'radio', 'boot', 'recovery', 'system', 'userdata', - 'cache', 'vendor'] - self.assertListEqual(fastboot_utils.ALL_PARTITIONS, parts) + fastboot_utils.FastbootUtils('abc') class FastbootUtilsWaitForFastbootMode(FastbootUtilsTest): @@ -110,10 +112,123 @@ class FastbootUtilsWaitForFastbootMode(FastbootUtilsTest): self.fastboot.WaitForFastbootMode() -class FastbootUtilsEnableFastbootMode(FastbootUtilsTest): +class FastbootUtilsIsFastbootMode(FastbootUtilsTest): + def testIsFastbootMode_True(self): + self.assertEqual(True, self.fastboot.IsFastbootMode()) + + def testIsFastbootMode_False(self): + self.fastboot._serial = 'not' + _SERIAL + self.assertEqual(False, self.fastboot.IsFastbootMode()) + + +class FastbootUtils_supports_ab(FastbootUtilsTest): + def test_supports_ab_fastboot_True(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('slot-count'), '2')): + self.assertEqual(True, self.fastboot.supports_ab) + + def test_supports_ab_fastboot_False(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('slot-count'), '1')): + self.assertEqual(False, self.fastboot.supports_ab) + + def test_supports_ab_fastboot_FalseWithEmpty(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('slot-count'), '')): + self.assertEqual(False, self.fastboot.supports_ab) + + def test_supports_ab_fastboot_FalseWithError(self): + with self.assertCalls((self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('slot-count'), + self.FastbootCommandFailedError([], ''))): + self.assertEqual(False, self.fastboot.supports_ab) + + def test_supports_ab_device_True(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), False), + (self.call.fastboot._device.GetProp('ro.build.ab_update'), 'true')): + self.assertEqual(True, self.fastboot.supports_ab) + + def test_supports_ab_device_False(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), False), + (self.call.fastboot._device.GetProp('ro.build.ab_update'), '')): + self.assertEqual(False, self.fastboot.supports_ab) + + +class FastbootUtils_requires_dtbo(FastbootUtilsTest): + def test_requires_dtbo_fastboot_True(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('has-slot:dtbo'), 'yes')): + self.assertEqual(True, self.fastboot.requires_dtbo) + + def test_requires_dtbo_fastboot_FalseWithEmpty(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('has-slot:dtbo'), '')): + self.assertEqual(False, self.fastboot.requires_dtbo) + + def test_requires_dtbo_fastboot_FalseWithError(self): + with self.assertCalls((self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('has-slot:dtbo'), + self.FastbootCommandFailedError([], ''))): + self.assertEqual(False, self.fastboot.requires_dtbo) + + def test_requires_dtbo_device_True(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), False), + (self.call.fastboot._device.GetProp('ro.boot.dtbo_idx'), '1')): + self.assertEqual(True, self.fastboot.requires_dtbo) + + def test_requires_dtbo_device_False(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), False), + (self.call.fastboot._device.GetProp('ro.boot.dtbo_idx'), '')): + self.assertEqual(False, self.fastboot.requires_dtbo) + + +class FastbootUtils_requires_vbmeta(FastbootUtilsTest): + def test_requires_vbmeta_fastboot_True(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('has-slot:vbmeta'), 'yes')): + self.assertEqual(True, self.fastboot.requires_vbmeta) + + def test_requires_vbmeta_fastboot_FalseWithEmpty(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('has-slot:vbmeta'), '')): + self.assertEqual(False, self.fastboot.requires_vbmeta) + + def test_requires_vbmeta_fastboot_FalseWithError(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), True), + (self.call.fastboot.fastboot.GetVar('has-slot:vbmeta'), + self.FastbootCommandFailedError([], ''))): + self.assertEqual(False, self.fastboot.requires_vbmeta) + + def test_requires_vbmeta_device_True(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), False), + (self.call.fastboot._device.GetProp('ro.boot.vbmeta.digest'), + '1')): + self.assertEqual(True, self.fastboot.requires_vbmeta) + + def test_requires_vbmeta_device_False(self): + with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), False), + (self.call.fastboot._device.GetProp('ro.boot.vbmeta.digest'), '')): + self.assertEqual(False, self.fastboot.requires_vbmeta) + +class FastbootUtilsEnableFastbootMode(FastbootUtilsTest): def testEnableFastbootMode(self): with self.assertCalls( + (self.call.fastboot.IsFastbootMode(), False), self.call.fastboot._device.EnableRoot(), self.call.fastboot._device.adb.Reboot(to_bootloader=True), self.call.fastboot.WaitForFastbootMode()): @@ -121,11 +236,9 @@ class FastbootUtilsEnableFastbootMode(FastbootUtilsTest): class FastbootUtilsReboot(FastbootUtilsTest): - def testReboot_bootloader(self): - with self.assertCalls( - self.call.fastboot.fastboot.RebootBootloader(), - self.call.fastboot.WaitForFastbootMode()): + with self.assertCalls(self.call.fastboot.fastboot.RebootBootloader(), + self.call.fastboot.WaitForFastbootMode()): self.fastboot.Reboot(bootloader=True) def testReboot_normal(self): @@ -136,40 +249,69 @@ class FastbootUtilsReboot(FastbootUtilsTest): class FastbootUtilsFlashPartitions(FastbootUtilsTest): - def testFlashPartitions_wipe(self): - with self.assertCalls( - (self.call.fastboot._VerifyBoard('test'), True), - (mock.call.devil.android.fastboot_utils. - _FindAndVerifyPartitionsAndImages(_PARTITIONS, 'test'), _IMAGES), - (self.call.fastboot.fastboot.Flash('bootloader', 'bootloader.img')), - (self.call.fastboot.Reboot(bootloader=True)), - (self.call.fastboot.fastboot.Flash('radio', 'radio.img')), - (self.call.fastboot.Reboot(bootloader=True)), - (self.call.fastboot.fastboot.Flash('boot', 'boot.img')), - (self.call.fastboot.fastboot.Flash('recovery', 'recovery.img')), - (self.call.fastboot.fastboot.Flash('system', 'system.img')), - (self.call.fastboot.fastboot.Flash('userdata', 'userdata.img')), - (self.call.fastboot.fastboot.Flash('cache', 'cache.img'))): - self.fastboot._FlashPartitions(_PARTITIONS, 'test', wipe=True) + with self.patch_call(self.call.fastboot.supports_ab, return_value=False): + with self.assertCalls( + (self.call.fastboot._VerifyBoard('test'), True), + (self.call.fastboot._FindAndVerifyPartitionsAndImages( + _PARTITIONS, 'test'), _IMAGES), + (self.call.fastboot.fastboot.Flash('bootloader', 'bootloader.img')), + (self.call.fastboot.Reboot(bootloader=True)), + (self.call.fastboot.fastboot.Flash('radio', 'radio.img')), + (self.call.fastboot.Reboot(bootloader=True)), + (self.call.fastboot.fastboot.Flash('boot', 'boot.img')), + (self.call.fastboot.fastboot.Flash('recovery', 'recovery.img')), + (self.call.fastboot.fastboot.Flash('system', 'system.img')), + (self.call.fastboot.fastboot.Flash('userdata', 'userdata.img')), + (self.call.fastboot.fastboot.Flash('cache', 'cache.img'))): + self.fastboot._FlashPartitions(_PARTITIONS, 'test', wipe=True) def testFlashPartitions_noWipe(self): - with self.assertCalls( - (self.call.fastboot._VerifyBoard('test'), True), - (mock.call.devil.android.fastboot_utils. - _FindAndVerifyPartitionsAndImages(_PARTITIONS, 'test'), _IMAGES), - (self.call.fastboot.fastboot.Flash('bootloader', 'bootloader.img')), - (self.call.fastboot.Reboot(bootloader=True)), - (self.call.fastboot.fastboot.Flash('radio', 'radio.img')), - (self.call.fastboot.Reboot(bootloader=True)), - (self.call.fastboot.fastboot.Flash('boot', 'boot.img')), - (self.call.fastboot.fastboot.Flash('recovery', 'recovery.img')), - (self.call.fastboot.fastboot.Flash('system', 'system.img'))): - self.fastboot._FlashPartitions(_PARTITIONS, 'test') + with self.patch_call(self.call.fastboot.supports_ab, return_value=False): + with self.assertCalls( + (self.call.fastboot._VerifyBoard('test'), True), + (self.call.fastboot._FindAndVerifyPartitionsAndImages( + _PARTITIONS, 'test'), _IMAGES), + (self.call.fastboot.fastboot.Flash('bootloader', 'bootloader.img')), + (self.call.fastboot.Reboot(bootloader=True)), + (self.call.fastboot.fastboot.Flash('radio', 'radio.img')), + (self.call.fastboot.Reboot(bootloader=True)), + (self.call.fastboot.fastboot.Flash('boot', 'boot.img')), + (self.call.fastboot.fastboot.Flash('recovery', 'recovery.img')), + (self.call.fastboot.fastboot.Flash('system', 'system.img'))): + self.fastboot._FlashPartitions(_PARTITIONS, 'test') + + def testFlashPartitions_AB_device(self): + ab_images = _IMAGES.copy() + ab_images['dtbo'] = 'dtbo.img' + ab_images['vbmeta'] = 'vbmeta.img' + ab_partitions = _PARTITIONS[:] + ab_partitions.append('dtbo') + ab_partitions.append('vbmeta') + with self.patch_call(self.call.fastboot.supports_ab, return_value=True): + with self.patch_call(self.call.fastboot.requires_dtbo, return_value=True): + with self.patch_call(self.call.fastboot.requires_vbmeta, + return_value=True): + with self.assertCalls( + (self.call.fastboot._VerifyBoard('test'), True), + (self.call.fastboot._FindAndVerifyPartitionsAndImages( + ab_partitions, 'test'), ab_images), + (self.call.fastboot.fastboot.Flash('bootloader', + 'bootloader.img')), + (self.call.fastboot.Reboot(bootloader=True)), + (self.call.fastboot.fastboot.Flash('radio', 'radio.img')), + (self.call.fastboot.Reboot(bootloader=True)), + (self.call.fastboot.fastboot.Flash('boot', 'boot.img')), + (self.call.fastboot.fastboot.Flash('recovery', 'recovery.img')), + (self.call.fastboot.fastboot.Flash('system', 'system.img')), + (self.call.fastboot.fastboot.Flash('userdata', 'userdata.img')), + (self.call.fastboot.fastboot.Flash('cache', 'cache.img')), + (self.call.fastboot.fastboot.Flash('dtbo', 'dtbo.img')), + (self.call.fastboot.fastboot.Flash('vbmeta', 'vbmeta.img'))): + self.fastboot._FlashPartitions(ab_partitions, 'test', wipe=True) class FastbootUtilsFastbootMode(FastbootUtilsTest): - def testFastbootMode_goodWait(self): with self.assertCalls( self.call.fastboot.EnableFastbootMode(), @@ -191,9 +333,7 @@ class FastbootUtilsFastbootMode(FastbootUtilsTest): def testFastbootMode_exception(self): with self.assertCalls( self.call.fastboot.EnableFastbootMode(), - self.call.fastboot.fastboot.SetOemOffModeCharge(False), - self.call.fastboot.fastboot.SetOemOffModeCharge(True), - self.call.fastboot.Reboot(wait_for_reboot=True)): + self.call.fastboot.fastboot.SetOemOffModeCharge(False)): with self.assertRaises(NotImplementedError): with self.fastboot.FastbootMode() as fbm: self.assertEqual(self.fastboot, fbm) @@ -208,7 +348,6 @@ class FastbootUtilsFastbootMode(FastbootUtilsTest): class FastbootUtilsVerifyBoard(FastbootUtilsTest): - def testVerifyBoard_bothValid(self): mock_file = io.StringIO(u'require board=%s\n' % _BOARD) with mock.patch('__builtin__.open', return_value=mock_file, create=True): @@ -257,37 +396,29 @@ class FastbootUtilsVerifyBoard(FastbootUtilsTest): class FastbootUtilsFindAndVerifyPartitionsAndImages(FastbootUtilsTest): - def testFindAndVerifyPartitionsAndImages_validNoVendor(self): PARTITIONS = [ 'bootloader', 'radio', 'boot', 'recovery', 'system', 'userdata', 'cache', 'vendor' ] files = [ - 'bootloader-test-.img', - 'radio123.img', - 'boot.img', - 'recovery.img', - 'system.img', - 'userdata.img', - 'cache.img' + 'bootloader-test-.img', 'radio123.img', 'boot.img', 'recovery.img', + 'system.img', 'userdata.img', 'cache.img' ] img_check = collections.OrderedDict([ - ('bootloader', 'test/bootloader-test-.img'), - ('radio', 'test/radio123.img'), - ('boot', 'test/boot.img'), - ('recovery', 'test/recovery.img'), - ('system', 'test/system.img'), - ('userdata', 'test/userdata.img'), - ('cache', 'test/cache.img'), + ('bootloader', 'test/bootloader-test-.img'), + ('radio', 'test/radio123.img'), + ('boot', 'test/boot.img'), + ('recovery', 'test/recovery.img'), + ('system', 'test/system.img'), + ('userdata', 'test/userdata.img'), + ('cache', 'test/cache.img'), ]) parts_check = [ - 'bootloader', 'radio', 'boot', 'recovery', 'system', 'userdata', - 'cache' + 'bootloader', 'radio', 'boot', 'recovery', 'system', 'userdata', 'cache' ] with mock.patch('os.listdir', return_value=files): - imgs = fastboot_utils._FindAndVerifyPartitionsAndImages( - PARTITIONS, 'test') + imgs = self.fastboot._FindAndVerifyPartitionsAndImages(PARTITIONS, 'test') parts = imgs.keys() self.assertDictEqual(imgs, img_check) self.assertListEqual(parts, parts_check) @@ -298,24 +429,18 @@ class FastbootUtilsFindAndVerifyPartitionsAndImages(FastbootUtilsTest): 'cache', 'vendor' ] files = [ - 'bootloader-test-.img', - 'radio123.img', - 'boot.img', - 'recovery.img', - 'system.img', - 'userdata.img', - 'cache.img', - 'vendor.img' + 'bootloader-test-.img', 'radio123.img', 'boot.img', 'recovery.img', + 'system.img', 'userdata.img', 'cache.img', 'vendor.img' ] img_check = { - 'bootloader': 'test/bootloader-test-.img', - 'radio': 'test/radio123.img', - 'boot': 'test/boot.img', - 'recovery': 'test/recovery.img', - 'system': 'test/system.img', - 'userdata': 'test/userdata.img', - 'cache': 'test/cache.img', - 'vendor': 'test/vendor.img', + 'bootloader': 'test/bootloader-test-.img', + 'radio': 'test/radio123.img', + 'boot': 'test/boot.img', + 'recovery': 'test/recovery.img', + 'system': 'test/system.img', + 'userdata': 'test/userdata.img', + 'cache': 'test/cache.img', + 'vendor': 'test/vendor.img', } parts_check = [ 'bootloader', 'radio', 'boot', 'recovery', 'system', 'userdata', @@ -323,25 +448,58 @@ class FastbootUtilsFindAndVerifyPartitionsAndImages(FastbootUtilsTest): ] with mock.patch('os.listdir', return_value=files): - imgs = fastboot_utils._FindAndVerifyPartitionsAndImages( - PARTITIONS, 'test') - parts = imgs.keys() - self.assertDictEqual(imgs, img_check) - self.assertListEqual(parts, parts_check) + with self.patch_call(self.call.fastboot.supports_ab, return_value=False): + imgs = self.fastboot._FindAndVerifyPartitionsAndImages( + PARTITIONS, 'test') + parts = imgs.keys() + self.assertDictEqual(imgs, img_check) + self.assertListEqual(parts, parts_check) def testFindAndVerifyPartitionsAndImages_badPartition(self): with mock.patch('os.listdir', return_value=['test']): - with self.assertRaises(KeyError): - fastboot_utils._FindAndVerifyPartitionsAndImages(['test'], 'test') + with self.patch_call(self.call.fastboot.supports_ab, return_value=False): + with self.assertRaises(KeyError): + self.fastboot._FindAndVerifyPartitionsAndImages(['test'], 'test') - def testFindAndVerifyPartitionsAndImages_noFile(self): + def testFindAndVerifyPartitionsAndImages_noFile_RequiredImage(self): with mock.patch('os.listdir', return_value=['test']): - with self.assertRaises(device_errors.FastbootCommandFailedError): - fastboot_utils._FindAndVerifyPartitionsAndImages(['cache'], 'test') + with self.patch_call(self.call.fastboot.supports_ab, return_value=False): + with self.assertRaises(device_errors.FastbootCommandFailedError): + self.fastboot._FindAndVerifyPartitionsAndImages(['boot'], 'test') + def testFindAndVerifyPartitionsAndImages_noFile_RequiredImageAB(self): + with mock.patch('os.listdir', return_value=['test']): + with self.patch_call(self.call.fastboot.supports_ab, return_value=True): + with self.assertRaises(device_errors.FastbootCommandFailedError): + self.fastboot._FindAndVerifyPartitionsAndImages(['boot'], 'test') -class FastbootUtilsFlashDevice(FastbootUtilsTest): + def testFindAndVerifyPartitionsAndImages_noFile_NotRequiredImage(self): + with mock.patch('os.listdir', return_value=['test']): + with self.patch_call(self.call.fastboot.supports_ab, return_value=False): + with self.patch_call(self.call.fastboot.requires_dtbo, + return_value=False): + self.assertFalse( + self.fastboot._FindAndVerifyPartitionsAndImages(['vendor'], + 'test')) + self.assertFalse( + self.fastboot._FindAndVerifyPartitionsAndImages(['dtbo'], 'test')) + + def testFindAndVerifyPartitionsAndImages_noFile_NotRequiredImageAB(self): + with mock.patch('os.listdir', return_value=['test']): + with self.patch_call(self.call.fastboot.supports_ab, return_value=True): + with self.patch_call(self.call.fastboot.requires_dtbo, + return_value=False): + self.assertFalse( + self.fastboot._FindAndVerifyPartitionsAndImages(['vendor'], + 'test')) + self.assertFalse( + self.fastboot._FindAndVerifyPartitionsAndImages(['cache'], + 'test')) + self.assertFalse( + self.fastboot._FindAndVerifyPartitionsAndImages(['dtbo'], 'test')) + +class FastbootUtilsFlashDevice(FastbootUtilsTest): def testFlashDevice_wipe(self): with self.assertCalls( self.call.fastboot.EnableFastbootMode(), diff --git a/catapult/devil/devil/android/flag_changer.py b/catapult/devil/devil/android/flag_changer.py index 110cf827..5dabc4c8 100644 --- a/catapult/devil/devil/android/flag_changer.py +++ b/catapult/devil/devil/android/flag_changer.py @@ -9,10 +9,8 @@ import re from devil.android.sdk import version_codes - logger = logging.getLogger(__name__) - _CMDLINE_DIR = '/data/local/tmp' _CMDLINE_DIR_LEGACY = '/data/local' _RE_NEEDS_QUOTING = re.compile(r'[^\w-]') # Not in: alphanumeric or hyphens. @@ -72,15 +70,15 @@ class FlagChanger(object): alternate_cmdline_path = posixpath.join(_CMDLINE_DIR_LEGACY, cmdline_file) if use_legacy_path: - cmdline_path, alternate_cmdline_path = ( - alternate_cmdline_path, cmdline_path) + cmdline_path, alternate_cmdline_path = (alternate_cmdline_path, + cmdline_path) if not self._device.HasRoot(): raise ValueError('use_legacy_path requires a rooted device') self._cmdline_path = cmdline_path if self._device.PathExists(alternate_cmdline_path): - logger.warning( - 'Removing alternate command line file %r.', alternate_cmdline_path) + logger.warning('Removing alternate command line file %r.', + alternate_cmdline_path) self._device.RemovePath(alternate_cmdline_path, as_root=True) self._state_stack = [None] # Actual state is set by GetCurrentFlags(). @@ -189,8 +187,8 @@ class FlagChanger(object): # permissions are needed, and document them in the code. if not self._device.HasRoot(): return - if (self._device.build_version_sdk >= version_codes.NOUGAT and - self._device.GetEnforce()): + if (self._device.build_version_sdk >= version_codes.NOUGAT + and self._device.GetEnforce()): self._device.SetEnforce(enabled=False) self._should_reset_enforce = True diff --git a/catapult/devil/devil/android/flag_changer_devicetest.py b/catapult/devil/devil/android/flag_changer_devicetest.py index b75504b5..4926ae3a 100644 --- a/catapult/devil/devil/android/flag_changer_devicetest.py +++ b/catapult/devil/devil/android/flag_changer_devicetest.py @@ -14,19 +14,21 @@ import unittest if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', ))) + os.path.abspath(os.path.join( + os.path.dirname(__file__), + '..', + '..', + ))) from devil.android import device_test_case from devil.android import device_utils from devil.android import flag_changer from devil.android.sdk import adb_wrapper - _CMDLINE_FILE = 'dummy-command-line' class FlagChangerTest(device_test_case.DeviceTestCase): - def setUp(self): super(FlagChangerTest, self).setUp() self.adb = adb_wrapper.AdbWrapper(self.serial) @@ -35,21 +37,21 @@ class FlagChangerTest(device_test_case.DeviceTestCase): self.adb, default_timeout=10, default_retries=0) # pylint: disable=protected-access self.cmdline_path = posixpath.join(flag_changer._CMDLINE_DIR, _CMDLINE_FILE) - self.cmdline_path_legacy = posixpath.join( - flag_changer._CMDLINE_DIR_LEGACY, _CMDLINE_FILE) + self.cmdline_path_legacy = posixpath.join(flag_changer._CMDLINE_DIR_LEGACY, + _CMDLINE_FILE) def tearDown(self): super(FlagChangerTest, self).tearDown() - self.device.RemovePath( - [self.cmdline_path, self.cmdline_path_legacy], force=True, as_root=True) + self.device.RemovePath([self.cmdline_path, self.cmdline_path_legacy], + force=True, + as_root=True) def testFlagChanger_restoreFlags(self): if not self.device.HasRoot(): self.skipTest('Test needs a rooted device') # Write some custom chrome command line flags. - self.device.WriteFile( - self.cmdline_path, 'chrome --some --old --flags') + self.device.WriteFile(self.cmdline_path, 'chrome --some --old --flags') # Write some more flags on a command line file in the legacy location. self.device.WriteFile( @@ -64,21 +66,17 @@ class FlagChangerTest(device_test_case.DeviceTestCase): # Write some new files, and check they are set. new_flags = ['--my', '--new', '--flags=with special value'] - self.assertItemsEqual( - changer.ReplaceFlags(new_flags), - new_flags) + self.assertItemsEqual(changer.ReplaceFlags(new_flags), new_flags) # Restore and go back to the old flags. - self.assertItemsEqual( - changer.Restore(), - ['--some', '--old', '--flags']) + self.assertItemsEqual(changer.Restore(), ['--some', '--old', '--flags']) def testFlagChanger_removeFlags(self): self.device.RemovePath(self.cmdline_path, force=True) self.assertFalse(self.device.PathExists(self.cmdline_path)) - with flag_changer.CustomCommandLineFlags( - self.device, _CMDLINE_FILE, ['--some', '--flags']): + with flag_changer.CustomCommandLineFlags(self.device, _CMDLINE_FILE, + ['--some', '--flags']): self.assertTrue(self.device.PathExists(self.cmdline_path)) self.assertFalse(self.device.PathExists(self.cmdline_path)) diff --git a/catapult/devil/devil/android/flag_changer_test.py b/catapult/devil/devil/android/flag_changer_test.py index dbe6facc..564ead6e 100755 --- a/catapult/devil/devil/android/flag_changer_test.py +++ b/catapult/devil/devil/android/flag_changer_test.py @@ -8,7 +8,6 @@ import unittest from devil.android import flag_changer - _CMDLINE_FILE = 'chrome-command-line' @@ -39,8 +38,8 @@ class FlagChangerTest(unittest.TestCase): self.device = _FakeDevice() # pylint: disable=protected-access self.cmdline_path = posixpath.join(flag_changer._CMDLINE_DIR, _CMDLINE_FILE) - self.cmdline_path_legacy = posixpath.join( - flag_changer._CMDLINE_DIR_LEGACY, _CMDLINE_FILE) + self.cmdline_path_legacy = posixpath.join(flag_changer._CMDLINE_DIR_LEGACY, + _CMDLINE_FILE) def testFlagChanger_removeAlternateCmdLine(self): self.device.WriteFile(self.cmdline_path_legacy, 'chrome --old --stuff') @@ -56,11 +55,11 @@ class FlagChangerTest(unittest.TestCase): self.device.WriteFile(self.cmdline_path, 'chrome --old --stuff') self.assertTrue(self.device.PathExists(self.cmdline_path)) - changer = flag_changer.FlagChanger(self.device, 'chrome-command-line', - use_legacy_path=True) + changer = flag_changer.FlagChanger( + self.device, 'chrome-command-line', use_legacy_path=True) self.assertEquals( - changer._cmdline_path, # pylint: disable=protected-access - self.cmdline_path_legacy) + changer._cmdline_path, # pylint: disable=protected-access + self.cmdline_path_legacy) self.assertFalse(self.device.PathExists(self.cmdline_path)) def testFlagChanger_mustBeFileName(self): @@ -89,21 +88,20 @@ class ParseSerializeFlagsTest(unittest.TestCase): self._testQuoteFlag('--key=valueA valueB', '--key="valueA valueB"') def testQuoteFlag_withQuotedValue2(self): - self._testQuoteFlag( - '--key=this "should" work', r'--key="this \"should\" work"') + self._testQuoteFlag('--key=this "should" work', + r'--key="this \"should\" work"') def testQuoteFlag_withQuotedValue3(self): - self._testQuoteFlag( - "--key=this is 'fine' too", '''--key="this is 'fine' too"''') + self._testQuoteFlag("--key=this is 'fine' too", + '''--key="this is 'fine' too"''') def testQuoteFlag_withQuotedValue4(self): - self._testQuoteFlag( - "--key='I really want to keep these quotes'", - '''--key="'I really want to keep these quotes'"''') + self._testQuoteFlag("--key='I really want to keep these quotes'", + '''--key="'I really want to keep these quotes'"''') def testQuoteFlag_withQuotedValue5(self): - self._testQuoteFlag( - "--this is a strange=flag", '"--this is a strange=flag"') + self._testQuoteFlag("--this is a strange=flag", + '"--this is a strange=flag"') def testQuoteFlag_withEmptyValue(self): self._testQuoteFlag('--some-flag=', '--some-flag=') @@ -122,24 +120,22 @@ class ParseSerializeFlagsTest(unittest.TestCase): self.assertItemsEqual(new_flags, expected_flags) def testParseCmdLine_simple(self): - self._testParseCmdLine( - 'chrome --foo --bar="a b" --baz=true --fine="ok"', - ['--foo', '--bar=a b', '--baz=true', '--fine=ok']) + self._testParseCmdLine('chrome --foo --bar="a b" --baz=true --fine="ok"', + ['--foo', '--bar=a b', '--baz=true', '--fine=ok']) def testParseCmdLine_withFancyQuotes(self): self._testParseCmdLine( r'''_ --foo="this 'is' ok" --bar='this \'is\' too' --baz="this \'is\' tricky" - ''', - ["--foo=this 'is' ok", - "--bar=this 'is' too", - r"--baz=this \'is\' tricky"]) + ''', [ + "--foo=this 'is' ok", "--bar=this 'is' too", + r"--baz=this \'is\' tricky" + ]) def testParseCmdLine_withUnterminatedQuote(self): - self._testParseCmdLine( - '_ --foo --bar="I forgot something', - ['--foo', '--bar=I forgot something']) + self._testParseCmdLine('_ --foo --bar="I forgot something', + ['--foo', '--bar=I forgot something']) if __name__ == '__main__': diff --git a/catapult/devil/devil/android/forwarder.py b/catapult/devil/devil/android/forwarder.py index 6be46516..cdb7e4d5 100644 --- a/catapult/devil/devil/android/forwarder.py +++ b/catapult/devil/devil/android/forwarder.py @@ -9,6 +9,8 @@ import inspect import logging import os import psutil +import re +import textwrap from devil import base_error from devil import devil_env @@ -25,6 +27,8 @@ logger = logging.getLogger(__name__) # Forwarder.DevicePortForHostPort. DYNAMIC_DEVICE_PORT = 0 +PORT_REGEX = re.compile(r'(?P<device_port>\d+):(?P<host_port>\d+)') + def _GetProcessStartTime(pid): p = psutil.Process(pid) @@ -34,7 +38,7 @@ def _GetProcessStartTime(pid): return p.create_time -def _LogMapFailureDiagnostics(device): +def _DumpHostLog(): # The host forwarder daemon logs to /tmp/host_forwarder_log, so print the end # of that. try: @@ -42,10 +46,14 @@ def _LogMapFailureDiagnostics(device): logger.info('Last 50 lines of the host forwarder daemon log:') for line in host_forwarder_log.read().splitlines()[-50:]: logger.info(' %s', line) - except Exception: # pylint: disable=broad-except + except Exception: # pylint: disable=broad-except # Grabbing the host forwarder log is best-effort. Ignore all errors. logger.warning('Failed to get the contents of host_forwarder_log.') + +def _LogMapFailureDiagnostics(device): + _DumpHostLog() + # The device forwarder daemon logs to the logcat, so print the end of that. try: logger.info('Last 50 lines of logcat:') @@ -101,10 +109,9 @@ class HostForwarderError(base_error.BaseError): class Forwarder(object): """Thread-safe class to manage port forwards from the device to the host.""" - _DEVICE_FORWARDER_FOLDER = (file_system.TEST_EXECUTABLE_DIR + - '/forwarder/') - _DEVICE_FORWARDER_PATH = (file_system.TEST_EXECUTABLE_DIR + - '/forwarder/device_forwarder') + _DEVICE_FORWARDER_FOLDER = (file_system.TEST_EXECUTABLE_DIR + '/forwarder/') + _DEVICE_FORWARDER_PATH = ( + file_system.TEST_EXECUTABLE_DIR + '/forwarder/device_forwarder') _LOCK_PATH = '/tmp/chrome.forwarder.lock' # Defined in host_forwarder_main.cc _HOST_FORWARDER_LOG = '/tmp/host_forwarder_log' @@ -137,11 +144,12 @@ class Forwarder(object): instance._InitDeviceLocked(device, tool) device_serial = str(device) - map_arg_lists = [ - ['--adb=' + adb_wrapper.AdbWrapper.GetAdbPath(), - '--serial-id=' + device_serial, - '--map', str(device_port), str(host_port)] - for device_port, host_port in port_pairs] + map_arg_lists = [[ + '--adb=' + adb_wrapper.AdbWrapper.GetAdbPath(), + '--serial-id=' + device_serial, '--map', + str(device_port), + str(host_port) + ] for device_port, host_port in port_pairs] logger.info('Forwarding using commands: %s', map_arg_lists) for map_arg_list in map_arg_lists: @@ -154,10 +162,10 @@ class Forwarder(object): '`%s` timed out:\n%s' % (' '.join(map_cmd), e.output)) except OSError as e: if e.errno == 2: - raise HostForwarderError( - 'Unable to start host forwarder. ' - 'Make sure you have built host_forwarder.') - else: raise + raise HostForwarderError('Unable to start host forwarder. ' + 'Make sure you have built host_forwarder.') + else: + raise if exit_code != 0: try: instance._KillDeviceLocked(device, tool) @@ -169,25 +177,25 @@ class Forwarder(object): 'Failed to kill the device forwarder after map failure: %s', str(e)) _LogMapFailureDiagnostics(device) - formatted_output = ('\n'.join(output) if isinstance(output, list) - else output) + formatted_output = ('\n'.join(output) + if isinstance(output, list) else output) raise HostForwarderError( - '`%s` exited with %d:\n%s' % ( - ' '.join(map_cmd), - exit_code, - formatted_output)) - tokens = output.split(':') - if len(tokens) != 2: - raise HostForwarderError( - 'Unexpected host forwarder output "%s", ' - 'expected "device_port:host_port"' % output) - device_port = int(tokens[0]) - host_port = int(tokens[1]) + '`%s` exited with %d:\n%s' % (' '.join(map_cmd), exit_code, + formatted_output)) + for line in output.splitlines(): + match = PORT_REGEX.match(line) + if match: + break + if not match: + raise HostForwarderError('Unable to find device_port:host_port in ' + 'host forwarder output: %s' % output) + device_port = int(match.groupdict()['device_port']) + host_port = int(match.groupdict()['host_port']) serial_with_port = (device_serial, device_port) instance._device_to_host_port_map[serial_with_port] = host_port instance._host_to_device_port_map[host_port] = serial_with_port - logger.info('Forwarding device port: %d to host port: %d.', - device_port, host_port) + logger.info('Forwarding device port: %d to host port: %d.', device_port, + host_port) @staticmethod def UnmapDevicePort(device_port, device): @@ -213,8 +221,7 @@ class Forwarder(object): unmap_all_cmd = [ instance._host_forwarder_path, '--adb=%s' % adb_wrapper.AdbWrapper.GetAdbPath(), - '--serial-id=%s' % device.serial, - '--unmap-all' + '--serial-id=%s' % device.serial, '--unmap-all' ] try: exit_code, output = cmd_helper.GetCmdStatusAndOutputWithTimeout( @@ -224,7 +231,8 @@ class Forwarder(object): '`%s` timed out:\n%s' % (' '.join(unmap_all_cmd), e.output)) if exit_code != 0: error_msg = [ - '`%s` exited with %d' % (' '.join(unmap_all_cmd), exit_code)] + '`%s` exited with %d' % (' '.join(unmap_all_cmd), exit_code) + ] if isinstance(output, list): error_msg += output else: @@ -317,8 +325,8 @@ class Forwarder(object): unmap_cmd = [ instance._host_forwarder_path, '--adb=%s' % adb_wrapper.AdbWrapper.GetAdbPath(), - '--serial-id=%s' % serial, - '--unmap', str(device_port) + '--serial-id=%s' % serial, '--unmap', + str(device_port) ] try: (exit_code, output) = cmd_helper.GetCmdStatusAndOutputWithTimeout( @@ -327,11 +335,8 @@ class Forwarder(object): raise HostForwarderError( '`%s` timed out:\n%s' % (' '.join(unmap_cmd), e.output)) if exit_code != 0: - logger.error( - '`%s` exited with %d:\n%s', - ' '.join(unmap_cmd), - exit_code, - '\n'.join(output) if isinstance(output, list) else output) + logger.error('`%s` exited with %d:\n%s', ' '.join(unmap_cmd), exit_code, + '\n'.join(output) if isinstance(output, list) else output) @staticmethod def _GetPidForLock(): @@ -393,18 +398,18 @@ class Forwarder(object): 'forwarder_device', device=device) forwarder_device_path_on_device = ( Forwarder._DEVICE_FORWARDER_FOLDER - if os.path.isdir(forwarder_device_path_on_host) - else Forwarder._DEVICE_FORWARDER_PATH) - device.PushChangedFiles([( - forwarder_device_path_on_host, - forwarder_device_path_on_device)]) + if os.path.isdir(forwarder_device_path_on_host) else + Forwarder._DEVICE_FORWARDER_PATH) + device.PushChangedFiles([(forwarder_device_path_on_host, + forwarder_device_path_on_device)]) cmd = [Forwarder._DEVICE_FORWARDER_PATH] wrapper = tool.GetUtilWrapper() if wrapper: cmd.insert(0, wrapper) device.RunShellCommand( - cmd, env={'LD_LIBRARY_PATH': Forwarder._DEVICE_FORWARDER_FOLDER}, + cmd, + env={'LD_LIBRARY_PATH': Forwarder._DEVICE_FORWARDER_FOLDER}, check_return=True) self._initialized_devices.add(device_serial) @@ -429,12 +434,41 @@ class Forwarder(object): kill_cmd = ['pkill', '-9', 'host_forwarder'] (exit_code, output) = cmd_helper.GetCmdStatusAndOutputWithTimeout( kill_cmd, Forwarder._TIMEOUT) - if exit_code != 0: - raise HostForwarderError( - '%s exited with %d:\n%s' % ( - self._host_forwarder_path, - exit_code, - '\n'.join(output) if isinstance(output, list) else output)) + if exit_code == -9: + # pkill can exit with -9, seemingly in cases where the process it's + # asked to kill dies sometime during pkill running. In this case, + # re-running should result in pkill succeeding. + logging.warning( + 'pkilling host forwarder returned -9, retrying. Output: %s', + output) + exit_code, output = cmd_helper.GetCmdStatusAndOutputWithTimeout( + kill_cmd, Forwarder._TIMEOUT) + if exit_code in (0, 1): + # pkill exits with a 0 if it was able to signal at least one process. + # pkill exits with a 1 if it wasn't able to singal a process because + # no matching process existed. We're ok with either. + return + + _, ps_output = cmd_helper.GetCmdStatusAndOutputWithTimeout( + ['ps', 'aux'], Forwarder._TIMEOUT) + host_forwarder_lines = [line for line in ps_output.splitlines() + if 'host_forwarder' in line] + if host_forwarder_lines: + logger.error('Remaining host_forwarder processes:\n %s', + '\n '.join(host_forwarder_lines)) + else: + logger.error('No remaining host_forwarder processes?') + _DumpHostLog() + error_msg = textwrap.dedent("""\ + `{kill_cmd}` failed to kill host_forwarder. + exit_code: {exit_code} + output: + {output} + """) + raise HostForwarderError( + error_msg.format( + kill_cmd=' '.join(kill_cmd), exit_code=str(exit_code), + output='\n'.join(' %s' % l for l in output.splitlines()))) except cmd_helper.TimeoutError as e: raise HostForwarderError( '`%s` timed out:\n%s' % (' '.join(kill_cmd), e.output)) @@ -472,5 +506,6 @@ class Forwarder(object): if wrapper: cmd.insert(0, wrapper) device.RunShellCommand( - cmd, env={'LD_LIBRARY_PATH': Forwarder._DEVICE_FORWARDER_FOLDER}, + cmd, + env={'LD_LIBRARY_PATH': Forwarder._DEVICE_FORWARDER_FOLDER}, check_return=True) diff --git a/catapult/devil/devil/android/install_commands.py b/catapult/devil/devil/android/install_commands.py index c8da8696..c87fc9bd 100644 --- a/catapult/devil/devil/android/install_commands.py +++ b/catapult/devil/devil/android/install_commands.py @@ -13,11 +13,10 @@ BIN_DIR = '%s/bin' % file_system.TEST_EXECUTABLE_DIR _FRAMEWORK_DIR = '%s/framework' % file_system.TEST_EXECUTABLE_DIR _COMMANDS = { - 'unzip': 'org.chromium.android.commands.unzip.Unzip', + 'unzip': 'org.chromium.android.commands.unzip.Unzip', } -_SHELL_COMMAND_FORMAT = ( -"""#!/system/bin/sh +_SHELL_COMMAND_FORMAT = ("""#!/system/bin/sh base=%s export CLASSPATH=$base/framework/chromium_commands.jar exec app_process $base/bin %s $@ @@ -39,19 +38,17 @@ def InstallCommands(device): chromium_commands_jar_path = devil_env.config.FetchPath('chromium_commands') if not os.path.exists(chromium_commands_jar_path): raise device_errors.CommandFailedError( - '%s not found. Please build chromium_commands.' - % chromium_commands_jar_path) + '%s not found. Please build chromium_commands.' % + chromium_commands_jar_path) - device.RunShellCommand( - ['mkdir', '-p', BIN_DIR, _FRAMEWORK_DIR], check_return=True) + device.RunShellCommand(['mkdir', '-p', BIN_DIR, _FRAMEWORK_DIR], + check_return=True) for command, main_class in _COMMANDS.iteritems(): - shell_command = _SHELL_COMMAND_FORMAT % ( - file_system.TEST_EXECUTABLE_DIR, main_class) + shell_command = _SHELL_COMMAND_FORMAT % (file_system.TEST_EXECUTABLE_DIR, + main_class) shell_file = '%s/%s' % (BIN_DIR, command) device.WriteFile(shell_file, shell_command) - device.RunShellCommand( - ['chmod', '755', shell_file], check_return=True) + device.RunShellCommand(['chmod', '755', shell_file], check_return=True) - device.adb.Push( - chromium_commands_jar_path, - '%s/chromium_commands.jar' % _FRAMEWORK_DIR) + device.adb.Push(chromium_commands_jar_path, + '%s/chromium_commands.jar' % _FRAMEWORK_DIR) diff --git a/catapult/devil/devil/android/logcat_monitor.py b/catapult/devil/devil/android/logcat_monitor.py index b5f796b7..bec74440 100644 --- a/catapult/devil/devil/android/logcat_monitor.py +++ b/catapult/devil/devil/android/logcat_monitor.py @@ -23,15 +23,20 @@ logger = logging.getLogger(__name__) class LogcatMonitor(object): - _RECORD_ITER_TIMEOUT = 0.2 + _RECORD_ITER_TIMEOUT = 0.01 _RECORD_THREAD_JOIN_WAIT = 5.0 _WAIT_TIME = 0.2 THREADTIME_RE_FORMAT = ( r'(?P<date>\S*) +(?P<time>\S*) +(?P<proc_id>%s) +(?P<thread_id>%s) +' r'(?P<log_level>%s) +(?P<component>%s) *: +(?P<message>%s)$') - def __init__(self, adb, clear=True, filter_specs=None, output_file=None, - transform_func=None, check_error=True): + def __init__(self, + adb, + clear=True, + filter_specs=None, + output_file=None, + transform_func=None, + check_error=True): """Create a LogcatMonitor instance. Args: @@ -63,7 +68,10 @@ class LogcatMonitor(object): return self._output_file @decorators.WithTimeoutAndRetriesDefaults(10, 0) - def WaitFor(self, success_regex, failure_regex=None, timeout=None, + def WaitFor(self, + success_regex, + failure_regex=None, + timeout=None, retries=None): """Wait for a matching logcat line or until a timeout occurs. @@ -118,7 +126,11 @@ class LogcatMonitor(object): else: time.sleep(self._WAIT_TIME) - def FindAll(self, message_regex, proc_id=None, thread_id=None, log_level=None, + def FindAll(self, + message_regex, + proc_id=None, + thread_id=None, + log_level=None, component=None): """Finds all lines in the logcat that match the provided constraints. @@ -153,8 +165,8 @@ class LogcatMonitor(object): component = r'[^\s:]+' # pylint: disable=protected-access threadtime_re = re.compile( - type(self).THREADTIME_RE_FORMAT % ( - proc_id, thread_id, log_level, component, message_regex)) + type(self).THREADTIME_RE_FORMAT % (proc_id, thread_id, log_level, + component, message_regex)) with open(self._record_file.name, 'r') as f: for line in f: @@ -168,6 +180,7 @@ class LogcatMonitor(object): Function spawns a thread that records logcat to file and will not die until |StopRecording| is called. """ + def record_to_file(): # Write the log with line buffering so the consumer sees each individual # line. @@ -259,8 +272,7 @@ class LogcatMonitor(object): """Closes logcat recording file in case |Close| was never called.""" with self._record_file_lock: if self._record_file: - logger.warning( - 'Need to call |Close| on the logcat monitor when done!') + logger.warning('Need to call |Close| on the logcat monitor when done!') self._record_file.close() @property diff --git a/catapult/devil/devil/android/logcat_monitor_test.py b/catapult/devil/devil/android/logcat_monitor_test.py index 8fb4d74b..7f2f10a6 100755 --- a/catapult/devil/devil/android/logcat_monitor_test.py +++ b/catapult/devil/devil/android/logcat_monitor_test.py @@ -28,19 +28,19 @@ class LogcatMonitorTest(unittest.TestCase): _TEST_THREADTIME_LOGCAT_DATA = [ '01-01 01:02:03.456 7890 0987 V LogcatMonitorTest: ' - 'verbose logcat monitor test message 1', + 'verbose logcat monitor test message 1', '01-01 01:02:03.457 8901 1098 D LogcatMonitorTest: ' - 'debug logcat monitor test message 2', + 'debug logcat monitor test message 2', '01-01 01:02:03.458 9012 2109 I LogcatMonitorTest: ' - 'info logcat monitor test message 3', + 'info logcat monitor test message 3', '01-01 01:02:03.459 0123 3210 W LogcatMonitorTest: ' - 'warning logcat monitor test message 4', + 'warning logcat monitor test message 4', '01-01 01:02:03.460 1234 4321 E LogcatMonitorTest: ' - 'error logcat monitor test message 5', + 'error logcat monitor test message 5', '01-01 01:02:03.461 2345 5432 F LogcatMonitorTest: ' - 'fatal logcat monitor test message 6', + 'fatal logcat monitor test message 6', '01-01 01:02:03.462 3456 6543 D LogcatMonitorTest: ' - 'last line' + 'last line' ] def assertIterEqual(self, expected_iter, actual_iter): @@ -71,8 +71,7 @@ class LogcatMonitorTest(unittest.TestCase): self.assertTrue(actual_match) self.assertEqual( '01-01 01:02:03.460 1234 4321 E LogcatMonitorTest: ' - 'error logcat monitor test message 5', - actual_match.group(0)) + 'error logcat monitor test message 5', actual_match.group(0)) self.assertEqual('error', actual_match.group(1)) test_log.Stop() test_log.Close() @@ -82,8 +81,8 @@ class LogcatMonitorTest(unittest.TestCase): test_log = _CreateTestLog( raw_logcat=type(self)._TEST_THREADTIME_LOGCAT_DATA) test_log.Start() - actual_match = test_log.WaitFor( - r'.*My Success Regex.*', r'.*(fatal|error) logcat monitor.*') + actual_match = test_log.WaitFor(r'.*My Success Regex.*', + r'.*(fatal|error) logcat monitor.*') self.assertIsNone(actual_match) test_log.Stop() test_log.Close() @@ -120,19 +119,18 @@ class LogcatMonitorTest(unittest.TestCase): test_log.Start() test_log.WaitFor(r'.*last line.*', None) test_log.Stop() - expected_results = [ - ('7890', '0987', 'V', 'LogcatMonitorTest', - 'verbose logcat monitor test message 1'), - ('8901', '1098', 'D', 'LogcatMonitorTest', - 'debug logcat monitor test message 2'), - ('9012', '2109', 'I', 'LogcatMonitorTest', - 'info logcat monitor test message 3'), - ('0123', '3210', 'W', 'LogcatMonitorTest', - 'warning logcat monitor test message 4'), - ('1234', '4321', 'E', 'LogcatMonitorTest', - 'error logcat monitor test message 5'), - ('2345', '5432', 'F', 'LogcatMonitorTest', - 'fatal logcat monitor test message 6')] + expected_results = [('7890', '0987', 'V', 'LogcatMonitorTest', + 'verbose logcat monitor test message 1'), + ('8901', '1098', 'D', 'LogcatMonitorTest', + 'debug logcat monitor test message 2'), + ('9012', '2109', 'I', 'LogcatMonitorTest', + 'info logcat monitor test message 3'), + ('0123', '3210', 'W', 'LogcatMonitorTest', + 'warning logcat monitor test message 4'), + ('1234', '4321', 'E', 'LogcatMonitorTest', + 'error logcat monitor test message 5'), + ('2345', '5432', 'F', 'LogcatMonitorTest', + 'fatal logcat monitor test message 6')] actual_results = test_log.FindAll(r'\S* logcat monitor test message \d') self.assertIterEqual(iter(expected_results), actual_results) test_log.Close() @@ -158,9 +156,8 @@ class LogcatMonitorTest(unittest.TestCase): test_log.Stop() actual_results = test_log.FindAll( r'\S* logcat monitor test message \d', proc_id=1234) - expected_results = [ - ('1234', '4321', 'E', 'LogcatMonitorTest', - 'error logcat monitor test message 5')] + expected_results = [('1234', '4321', 'E', 'LogcatMonitorTest', + 'error logcat monitor test message 5')] self.assertIterEqual(iter(expected_results), actual_results) test_log.Close() @@ -173,9 +170,8 @@ class LogcatMonitorTest(unittest.TestCase): test_log.Stop() actual_results = test_log.FindAll( r'\S* logcat monitor test message \d', thread_id=2109) - expected_results = [ - ('9012', '2109', 'I', 'LogcatMonitorTest', - 'info logcat monitor test message 3')] + expected_results = [('9012', '2109', 'I', 'LogcatMonitorTest', + 'info logcat monitor test message 3')] self.assertIterEqual(iter(expected_results), actual_results) test_log.Close() @@ -188,12 +184,10 @@ class LogcatMonitorTest(unittest.TestCase): test_log.Stop() actual_results = test_log.FindAll( r'\S* logcat monitor test message \d', log_level=r'[DW]') - expected_results = [ - ('8901', '1098', 'D', 'LogcatMonitorTest', - 'debug logcat monitor test message 2'), - ('0123', '3210', 'W', 'LogcatMonitorTest', - 'warning logcat monitor test message 4') - ] + expected_results = [('8901', '1098', 'D', 'LogcatMonitorTest', + 'debug logcat monitor test message 2'), + ('0123', '3210', 'W', 'LogcatMonitorTest', + 'warning logcat monitor test message 4')] self.assertIterEqual(iter(expected_results), actual_results) test_log.Close() @@ -205,26 +199,22 @@ class LogcatMonitorTest(unittest.TestCase): test_log.WaitFor(r'.*last line.*', None) test_log.Stop() actual_results = test_log.FindAll(r'.*', component='LogcatMonitorTest') - expected_results = [ - ('7890', '0987', 'V', 'LogcatMonitorTest', - 'verbose logcat monitor test message 1'), - ('8901', '1098', 'D', 'LogcatMonitorTest', - 'debug logcat monitor test message 2'), - ('9012', '2109', 'I', 'LogcatMonitorTest', - 'info logcat monitor test message 3'), - ('0123', '3210', 'W', 'LogcatMonitorTest', - 'warning logcat monitor test message 4'), - ('1234', '4321', 'E', 'LogcatMonitorTest', - 'error logcat monitor test message 5'), - ('2345', '5432', 'F', 'LogcatMonitorTest', - 'fatal logcat monitor test message 6'), - ('3456', '6543', 'D', 'LogcatMonitorTest', - 'last line') - ] + expected_results = [('7890', '0987', 'V', 'LogcatMonitorTest', + 'verbose logcat monitor test message 1'), + ('8901', '1098', 'D', 'LogcatMonitorTest', + 'debug logcat monitor test message 2'), + ('9012', '2109', 'I', 'LogcatMonitorTest', + 'info logcat monitor test message 3'), + ('0123', '3210', 'W', 'LogcatMonitorTest', + 'warning logcat monitor test message 4'), + ('1234', '4321', 'E', 'LogcatMonitorTest', + 'error logcat monitor test message 5'), + ('2345', '5432', 'F', 'LogcatMonitorTest', + 'fatal logcat monitor test message 6'), + ('3456', '6543', 'D', 'LogcatMonitorTest', 'last line')] self.assertIterEqual(iter(expected_results), actual_results) test_log.Close() if __name__ == '__main__': unittest.main(verbosity=2) - diff --git a/catapult/devil/devil/android/md5sum.py b/catapult/devil/devil/android/md5sum.py index f5b6f3cf..8adf4ef7 100644 --- a/catapult/devil/devil/android/md5sum.py +++ b/catapult/devil/devil/android/md5sum.py @@ -2,9 +2,11 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +import base64 +import gzip import os -import posixpath import re +import StringIO from devil import devil_env from devil.android import device_errors @@ -13,7 +15,16 @@ from devil.utils import cmd_helper MD5SUM_DEVICE_LIB_PATH = '/data/local/tmp/md5sum' MD5SUM_DEVICE_BIN_PATH = MD5SUM_DEVICE_LIB_PATH + '/md5sum_bin' -_STARTS_WITH_CHECKSUM_RE = re.compile(r'^\s*[0-9a-fA-F]{32}\s+') +_STARTS_WITH_CHECKSUM_RE = re.compile(r'^[0-9a-fA-F]{16}$') + +# We need to cap how many paths we send to the md5_sum binaries at once because +# the ARG_MAX on Android devices is relatively small, typically 131072 bytes. +# However, the more paths we use per invocation, the lower the overhead of +# starting processes, so we want to maximize this number, but we can't compute +# it exactly as we don't know how well our paths will compress. +# 5000 is experimentally determined to be reasonable. 10000 fails, and 7500 +# works with existing usage, so 5000 seems like a pretty safe compromise. +_MAX_PATHS_PER_INVOCATION = 5000 def CalculateHostMd5Sums(paths): @@ -29,14 +40,22 @@ def CalculateHostMd5Sums(paths): """ if isinstance(paths, basestring): paths = [paths] + paths = list(paths) md5sum_bin_host_path = devil_env.config.FetchPath('md5sum_host') if not os.path.exists(md5sum_bin_host_path): raise IOError('File not built: %s' % md5sum_bin_host_path) - out = cmd_helper.GetCmdOutput( - [md5sum_bin_host_path] + [os.path.realpath(p) for p in paths]) + out = "" + for i in range(0, len(paths), _MAX_PATHS_PER_INVOCATION): + mem_file = StringIO.StringIO() + compressed = gzip.GzipFile(fileobj=mem_file, mode="wb") + compressed.write(";".join( + [os.path.realpath(p) for p in paths[i:i+_MAX_PATHS_PER_INVOCATION]])) + compressed.close() + compressed_paths = base64.b64encode(mem_file.getvalue()) + out += cmd_helper.GetCmdOutput([md5sum_bin_host_path, "-gz", compressed_paths]) - return _ParseMd5SumOutput(out.splitlines()) + return dict(zip(paths, out.splitlines())) def CalculateDeviceMd5Sums(paths, device): @@ -55,7 +74,6 @@ def CalculateDeviceMd5Sums(paths, device): if isinstance(paths, basestring): paths = [paths] - # Allow generators paths = list(paths) md5sum_dist_path = devil_env.config.FetchPath('md5sum_device', device=device) @@ -78,16 +96,13 @@ def CalculateDeviceMd5Sums(paths, device): md5sum_script += '! [[ $(ls -l $a) = *%d* ]]&&exit 2;' % md5sum_file_size # Make sure it can find libbase.so md5sum_script += 'export LD_LIBRARY_PATH=%s;' % MD5SUM_DEVICE_LIB_PATH - if len(paths) > 1: - prefix = posixpath.commonprefix(paths) - if len(prefix) > 4: - md5sum_script += 'p="%s";' % prefix - paths = ['$p"%s"' % p[len(prefix):] for p in paths] - - md5sum_script += ';'.join('$a %s' % p for p in paths) - # Don't fail the script if the last md5sum fails (due to file not found) - # Note: ":" is equivalent to "true". - md5sum_script += ';:' + for i in range(0, len(paths), _MAX_PATHS_PER_INVOCATION): + mem_file = StringIO.StringIO() + compressed = gzip.GzipFile(fileobj=mem_file, mode="wb") + compressed.write(";".join(paths[i:i+_MAX_PATHS_PER_INVOCATION])) + compressed.close() + compressed_paths = base64.b64encode(mem_file.getvalue()) + md5sum_script += '$a -gz %s;' % compressed_paths try: out = device.RunShellCommand( md5sum_script, shell=True, check_return=True, large_output=True) @@ -99,7 +114,8 @@ def CalculateDeviceMd5Sums(paths, device): # to re-push as non-root causes the push command to report success, but # actually fail. So, wipe the directory first. device.RunShellCommand(['rm', '-rf', MD5SUM_DEVICE_LIB_PATH], - as_root=True, check_return=True) + as_root=True, + check_return=True) if os.path.isdir(md5sum_dist_path): device.adb.Push(md5sum_dist_path, MD5SUM_DEVICE_LIB_PATH) else: @@ -112,11 +128,4 @@ def CalculateDeviceMd5Sums(paths, device): else: raise - return _ParseMd5SumOutput(out) - - -def _ParseMd5SumOutput(out): - hash_and_path = (l.split(None, 1) for l in out - if l and _STARTS_WITH_CHECKSUM_RE.match(l)) - return dict((p, h) for h, p in hash_and_path) - + return dict(zip(paths, [l for l in out if _STARTS_WITH_CHECKSUM_RE.match(l)])) diff --git a/catapult/devil/devil/android/md5sum_test.py b/catapult/devil/devil/android/md5sum_test.py index c9b49545..548b2d02 100755 --- a/catapult/devil/devil/android/md5sum_test.py +++ b/catapult/devil/devil/android/md5sum_test.py @@ -19,17 +19,16 @@ MD5_DIST = os.path.join(TEST_OUT_DIR, 'md5sum_dist') class Md5SumTest(unittest.TestCase): - def setUp(self): mocked_attrs = { - 'md5sum_host': HOST_MD5_EXECUTABLE, - 'md5sum_device': MD5_DIST, + 'md5sum_host': HOST_MD5_EXECUTABLE, + 'md5sum_device': MD5_DIST, } self._patchers = [ - mock.patch('devil.devil_env._Environment.FetchPath', - mock.Mock(side_effect=lambda a, device=None: mocked_attrs[a])), - mock.patch('os.path.exists', - new=mock.Mock(return_value=True)), + mock.patch( + 'devil.devil_env._Environment.FetchPath', + mock.Mock(side_effect=lambda a, device=None: mocked_attrs[a])), + mock.patch('os.path.exists', new=mock.Mock(return_value=True)), ] for p in self._patchers: p.start() @@ -41,53 +40,30 @@ class Md5SumTest(unittest.TestCase): def testCalculateHostMd5Sums_singlePath(self): test_path = '/test/host/file.dat' mock_get_cmd_output = mock.Mock( - return_value='0123456789abcdeffedcba9876543210 /test/host/file.dat') - with mock.patch('devil.utils.cmd_helper.GetCmdOutput', - new=mock_get_cmd_output): + return_value='0123456789abcdef') + with mock.patch( + 'devil.utils.cmd_helper.GetCmdOutput', new=mock_get_cmd_output): out = md5sum.CalculateHostMd5Sums(test_path) self.assertEquals(1, len(out)) self.assertTrue('/test/host/file.dat' in out) - self.assertEquals('0123456789abcdeffedcba9876543210', - out['/test/host/file.dat']) + self.assertEquals('0123456789abcdef', out['/test/host/file.dat']) mock_get_cmd_output.assert_called_once_with( - [HOST_MD5_EXECUTABLE, '/test/host/file.dat']) + [HOST_MD5_EXECUTABLE, "-gz", mock.ANY]) def testCalculateHostMd5Sums_list(self): test_paths = ['/test/host/file0.dat', '/test/host/file1.dat'] mock_get_cmd_output = mock.Mock( - return_value='0123456789abcdeffedcba9876543210 /test/host/file0.dat\n' - '123456789abcdef00fedcba987654321 /test/host/file1.dat\n') - with mock.patch('devil.utils.cmd_helper.GetCmdOutput', - new=mock_get_cmd_output): + return_value='0123456789abcdef\n123456789abcdef0\n') + with mock.patch( + 'devil.utils.cmd_helper.GetCmdOutput', new=mock_get_cmd_output): out = md5sum.CalculateHostMd5Sums(test_paths) self.assertEquals(2, len(out)) self.assertTrue('/test/host/file0.dat' in out) - self.assertEquals('0123456789abcdeffedcba9876543210', - out['/test/host/file0.dat']) + self.assertEquals('0123456789abcdef', out['/test/host/file0.dat']) self.assertTrue('/test/host/file1.dat' in out) - self.assertEquals('123456789abcdef00fedcba987654321', - out['/test/host/file1.dat']) + self.assertEquals('123456789abcdef0', out['/test/host/file1.dat']) mock_get_cmd_output.assert_called_once_with( - [HOST_MD5_EXECUTABLE, '/test/host/file0.dat', - '/test/host/file1.dat']) - - def testCalculateHostMd5Sums_generator(self): - test_paths = ('/test/host/' + p for p in ['file0.dat', 'file1.dat']) - mock_get_cmd_output = mock.Mock( - return_value='0123456789abcdeffedcba9876543210 /test/host/file0.dat\n' - '123456789abcdef00fedcba987654321 /test/host/file1.dat\n') - with mock.patch('devil.utils.cmd_helper.GetCmdOutput', - new=mock_get_cmd_output): - out = md5sum.CalculateHostMd5Sums(test_paths) - self.assertEquals(2, len(out)) - self.assertTrue('/test/host/file0.dat' in out) - self.assertEquals('0123456789abcdeffedcba9876543210', - out['/test/host/file0.dat']) - self.assertTrue('/test/host/file1.dat' in out) - self.assertEquals('123456789abcdef00fedcba987654321', - out['/test/host/file1.dat']) - mock_get_cmd_output.assert_called_once_with( - [HOST_MD5_EXECUTABLE, '/test/host/file0.dat', '/test/host/file1.dat']) + [HOST_MD5_EXECUTABLE, "-gz", mock.ANY]) def testCalculateDeviceMd5Sums_noPaths(self): device = mock.NonCallableMock() @@ -100,29 +76,26 @@ class Md5SumTest(unittest.TestCase): test_path = '/storage/emulated/legacy/test/file.dat' device = mock.NonCallableMock() - device_md5sum_output = [ - '0123456789abcdeffedcba9876543210 ' - '/storage/emulated/legacy/test/file.dat', - ] + device_md5sum_output = ['0123456789abcdef',] device.RunShellCommand = mock.Mock(return_value=device_md5sum_output) with mock.patch('os.path.getsize', return_value=1337): out = md5sum.CalculateDeviceMd5Sums(test_path, device) self.assertEquals(1, len(out)) self.assertTrue('/storage/emulated/legacy/test/file.dat' in out) - self.assertEquals('0123456789abcdeffedcba9876543210', + self.assertEquals('0123456789abcdef', out['/storage/emulated/legacy/test/file.dat']) self.assertEquals(1, len(device.RunShellCommand.call_args_list)) def testCalculateDeviceMd5Sums_list(self): - test_path = ['/storage/emulated/legacy/test/file0.dat', - '/storage/emulated/legacy/test/file1.dat'] + test_path = [ + '/storage/emulated/legacy/test/file0.dat', + '/storage/emulated/legacy/test/file1.dat' + ] device = mock.NonCallableMock() device_md5sum_output = [ - '0123456789abcdeffedcba9876543210 ' - '/storage/emulated/legacy/test/file0.dat', - '123456789abcdef00fedcba987654321 ' - '/storage/emulated/legacy/test/file1.dat', + '0123456789abcdef', + '123456789abcdef0', ] device.RunShellCommand = mock.Mock(return_value=device_md5sum_output) @@ -130,10 +103,10 @@ class Md5SumTest(unittest.TestCase): out = md5sum.CalculateDeviceMd5Sums(test_path, device) self.assertEquals(2, len(out)) self.assertTrue('/storage/emulated/legacy/test/file0.dat' in out) - self.assertEquals('0123456789abcdeffedcba9876543210', + self.assertEquals('0123456789abcdef', out['/storage/emulated/legacy/test/file0.dat']) self.assertTrue('/storage/emulated/legacy/test/file1.dat' in out) - self.assertEquals('123456789abcdef00fedcba987654321', + self.assertEquals('123456789abcdef0', out['/storage/emulated/legacy/test/file1.dat']) self.assertEquals(1, len(device.RunShellCommand.call_args_list)) @@ -143,10 +116,8 @@ class Md5SumTest(unittest.TestCase): device = mock.NonCallableMock() device_md5sum_output = [ - '0123456789abcdeffedcba9876543210 ' - '/storage/emulated/legacy/test/file0.dat', - '123456789abcdef00fedcba987654321 ' - '/storage/emulated/legacy/test/file1.dat', + '0123456789abcdef', + '123456789abcdef0', ] device.RunShellCommand = mock.Mock(return_value=device_md5sum_output) @@ -154,10 +125,10 @@ class Md5SumTest(unittest.TestCase): out = md5sum.CalculateDeviceMd5Sums(test_path, device) self.assertEquals(2, len(out)) self.assertTrue('/storage/emulated/legacy/test/file0.dat' in out) - self.assertEquals('0123456789abcdeffedcba9876543210', + self.assertEquals('0123456789abcdef', out['/storage/emulated/legacy/test/file0.dat']) self.assertTrue('/storage/emulated/legacy/test/file1.dat' in out) - self.assertEquals('123456789abcdef00fedcba987654321', + self.assertEquals('123456789abcdef0', out['/storage/emulated/legacy/test/file1.dat']) self.assertEquals(1, len(device.RunShellCommand.call_args_list)) @@ -168,10 +139,9 @@ class Md5SumTest(unittest.TestCase): device = mock.NonCallableMock() device_md5sum_output = [ 'WARNING: linker: /data/local/tmp/md5sum/md5sum_bin: ' - 'unused DT entry: type 0x1d arg 0x15db', + 'unused DT entry: type 0x1d arg 0x15db', 'THIS_IS_NOT_A_VALID_CHECKSUM_ZZZ some random text', - '0123456789abcdeffedcba9876543210 ' - '/storage/emulated/legacy/test/file.dat', + '0123456789abcdef', ] device.RunShellCommand = mock.Mock(return_value=device_md5sum_output) @@ -179,17 +149,18 @@ class Md5SumTest(unittest.TestCase): out = md5sum.CalculateDeviceMd5Sums(test_path, device) self.assertEquals(1, len(out)) self.assertTrue('/storage/emulated/legacy/test/file.dat' in out) - self.assertEquals('0123456789abcdeffedcba9876543210', + self.assertEquals('0123456789abcdef', out['/storage/emulated/legacy/test/file.dat']) self.assertEquals(1, len(device.RunShellCommand.call_args_list)) def testCalculateDeviceMd5Sums_list_fileMissing(self): - test_path = ['/storage/emulated/legacy/test/file0.dat', - '/storage/emulated/legacy/test/file1.dat'] + test_path = [ + '/storage/emulated/legacy/test/file0.dat', + '/storage/emulated/legacy/test/file1.dat' + ] device = mock.NonCallableMock() device_md5sum_output = [ - '0123456789abcdeffedcba9876543210 ' - '/storage/emulated/legacy/test/file0.dat', + '0123456789abcdef', '[0819/203513:ERROR:md5sum.cc(25)] Could not open file asdf', '', ] @@ -199,7 +170,7 @@ class Md5SumTest(unittest.TestCase): out = md5sum.CalculateDeviceMd5Sums(test_path, device) self.assertEquals(1, len(out)) self.assertTrue('/storage/emulated/legacy/test/file0.dat' in out) - self.assertEquals('0123456789abcdeffedcba9876543210', + self.assertEquals('0123456789abcdef', out['/storage/emulated/legacy/test/file0.dat']) self.assertEquals(1, len(device.RunShellCommand.call_args_list)) @@ -211,27 +182,26 @@ class Md5SumTest(unittest.TestCase): device.adb.Push = mock.Mock() device_md5sum_output = [ 'WARNING: linker: /data/local/tmp/md5sum/md5sum_bin: ' - 'unused DT entry: type 0x1d arg 0x15db', + 'unused DT entry: type 0x1d arg 0x15db', 'THIS_IS_NOT_A_VALID_CHECKSUM_ZZZ some random text', - '0123456789abcdeffedcba9876543210 ' - '/storage/emulated/legacy/test/file.dat', + '0123456789abcdef', ] error = device_errors.AdbShellCommandFailedError('cmd', 'out', 2) device.RunShellCommand = mock.Mock( side_effect=(error, '', device_md5sum_output)) - with mock.patch('os.path.isdir', return_value=True), ( - mock.patch('os.path.getsize', return_value=1337)): + with mock.patch( + 'os.path.isdir', return_value=True), (mock.patch( + 'os.path.getsize', return_value=1337)): out = md5sum.CalculateDeviceMd5Sums(test_path, device) self.assertEquals(1, len(out)) self.assertTrue('/storage/emulated/legacy/test/file.dat' in out) - self.assertEquals('0123456789abcdeffedcba9876543210', + self.assertEquals('0123456789abcdef', out['/storage/emulated/legacy/test/file.dat']) self.assertEquals(3, len(device.RunShellCommand.call_args_list)) - device.adb.Push.assert_called_once_with( - 'test/out/directory/md5sum_dist', '/data/local/tmp/md5sum') + device.adb.Push.assert_called_once_with('test/out/directory/md5sum_dist', + '/data/local/tmp/md5sum') if __name__ == '__main__': unittest.main(verbosity=2) - diff --git a/catapult/devil/devil/android/ndk/abis.py b/catapult/devil/devil/android/ndk/abis.py index dd32f7cb..e92ef2d1 100644 --- a/catapult/devil/devil/android/ndk/abis.py +++ b/catapult/devil/devil/android/ndk/abis.py @@ -1,7 +1,6 @@ # Copyright 2019 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Android NDK ABIs. https://developer.android.com/ndk/guides/abis diff --git a/catapult/devil/devil/android/perf/perf_control.py b/catapult/devil/devil/android/perf/perf_control.py index 1226be80..59485e0e 100644 --- a/catapult/devil/devil/android/perf/perf_control.py +++ b/catapult/devil/devil/android/perf/perf_control.py @@ -11,7 +11,6 @@ from devil.android import device_errors logger = logging.getLogger(__name__) _atexit_messages = set() - # Defines how to switch between the default performance configuration # ('default_mode') and the mode for use when benchmarking ('high_perf_mode'). # For devices not in the list the defaults are to set up the scaling governor to @@ -24,93 +23,138 @@ _atexit_messages = set() # TODO(crbug.com/383566): Add definitions for all devices used in the perf # waterfall. _PERFORMANCE_MODE_DEFINITIONS = { - # Fire TV Edition - 4K - 'AFTKMST12': { - 'default_mode_governor': 'interactive', - }, - # Pixel 3 - 'blueline': { - 'high_perf_mode': { - 'bring_cpu_cores_online': True, - # The SoC is Arm big.LITTLE. The cores 0..3 are LITTLE, the 4..7 are big. - 'cpu_max_freq': {'0..3': 1228800, '4..7': 1536000}, - 'gpu_max_freq': 520000000, - }, - 'default_mode': { - 'cpu_max_freq': {'0..3': 1766400, '4..7': 2649600}, - 'gpu_max_freq': 710000000, - }, - 'default_mode_governor': 'schedutil', - }, - 'GT-I9300': { - 'default_mode_governor': 'pegasusq', - }, - 'Galaxy Nexus': { - 'default_mode_governor': 'interactive', - }, - # Pixel - 'msm8996': { - 'high_perf_mode': { - 'bring_cpu_cores_online': True, - 'cpu_max_freq': 1209600, - 'gpu_max_freq': 315000000, - }, - 'default_mode': { - # The SoC is Arm big.LITTLE. The cores 0..1 are LITTLE, the 2..3 are big. - 'cpu_max_freq': {'0..1': 1593600, '2..3': 2150400}, - 'gpu_max_freq': 624000000, - }, - 'default_mode_governor': 'sched', - }, - 'Nexus 7': { - 'default_mode_governor': 'interactive', - }, - 'Nexus 10': { - 'default_mode_governor': 'interactive', - }, - 'Nexus 4': { - 'high_perf_mode': { - 'bring_cpu_cores_online': True, + # Fire TV Edition - 4K + 'AFTKMST12': { + 'default_mode_governor': 'interactive', + }, + # Pixel 3 + 'blueline': { + 'high_perf_mode': { + 'bring_cpu_cores_online': True, + # The SoC is Arm big.LITTLE. The cores 0..3 are LITTLE, + # the 4..7 are big. + 'cpu_max_freq': { + '0..3': 1228800, + '4..7': 1536000 + }, + 'gpu_max_freq': 520000000, + }, + 'default_mode': { + 'cpu_max_freq': { + '0..3': 1766400, + '4..7': 2649600 + }, + 'gpu_max_freq': 710000000, + }, + 'big_cores': ['4', '5', '6', '7'], + 'default_mode_governor': 'schedutil', + }, + 'Pixel 2': { + 'high_perf_mode': { + 'bring_cpu_cores_online': True, + # These are set to roughly 7/8 of the max frequency. The purpose of + # this is to ensure that thermal throttling doesn't kick in midway + # through a test and cause flaky results. It should also improve the + # longevity of the devices by keeping them cooler. + 'cpu_max_freq': { + '0..3': 1670400, + '4..7': 2208000, + }, + 'gpu_max_freq': 670000000, + }, + 'default_mode': { + # These are the maximum frequencies available for these CPUs and + # GPUs. + 'cpu_max_freq': { + '0..3': 1900800, + '4..7': 2457600, + }, + 'gpu_max_freq': 710000000, + }, + 'big_cores': ['4', '5', '6', '7'], + 'default_mode_governor': 'schedutil', + }, + 'GT-I9300': { + 'default_mode_governor': 'pegasusq', + }, + 'Galaxy Nexus': { + 'default_mode_governor': 'interactive', + }, + # Pixel + 'msm8996': { + 'high_perf_mode': { + 'bring_cpu_cores_online': True, + 'cpu_max_freq': 1209600, + 'gpu_max_freq': 315000000, + }, + 'default_mode': { + # The SoC is Arm big.LITTLE. The cores 0..1 are LITTLE, + # the 2..3 are big. + 'cpu_max_freq': { + '0..1': 1593600, + '2..3': 2150400 + }, + 'gpu_max_freq': 624000000, + }, + 'big_cores': ['2', '3'], + 'default_mode_governor': 'sched', }, - 'default_mode_governor': 'ondemand', - }, - 'Nexus 5': { - # The list of possible GPU frequency values can be found in: - # /sys/class/kgsl/kgsl-3d0/gpu_available_frequencies. - # For CPU cores the possible frequency values are at: - # /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies - 'high_perf_mode': { - 'bring_cpu_cores_online': True, - 'cpu_max_freq': 1190400, - 'gpu_max_freq': 200000000, + 'Nexus 7': { + 'default_mode_governor': 'interactive', }, - 'default_mode': { - 'cpu_max_freq': 2265600, - 'gpu_max_freq': 450000000, + 'Nexus 10': { + 'default_mode_governor': 'interactive', }, - 'default_mode_governor': 'ondemand', - }, - 'Nexus 5X': { - 'high_perf_mode': { - 'bring_cpu_cores_online': True, - 'cpu_max_freq': 1248000, - 'gpu_max_freq': 300000000, + 'Nexus 4': { + 'high_perf_mode': { + 'bring_cpu_cores_online': True, + }, + 'default_mode_governor': 'ondemand', }, - 'default_mode': { - 'governor': 'ondemand', - # The SoC is ARM big.LITTLE. The cores 4..5 are big, the 0..3 are LITTLE. - 'cpu_max_freq': {'0..3': 1440000, '4..5': 1824000}, - 'gpu_max_freq': 600000000, + 'Nexus 5': { + # The list of possible GPU frequency values can be found in: + # /sys/class/kgsl/kgsl-3d0/gpu_available_frequencies. + # For CPU cores the possible frequency values are at: + # /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies + 'high_perf_mode': { + 'bring_cpu_cores_online': True, + 'cpu_max_freq': 1190400, + 'gpu_max_freq': 200000000, + }, + 'default_mode': { + 'cpu_max_freq': 2265600, + 'gpu_max_freq': 450000000, + }, + 'default_mode_governor': 'ondemand', + }, + 'Nexus 5X': { + 'high_perf_mode': { + 'bring_cpu_cores_online': True, + 'cpu_max_freq': 1248000, + 'gpu_max_freq': 300000000, + }, + 'default_mode': { + 'governor': 'ondemand', + # The SoC is ARM big.LITTLE. The cores 4..5 are big, + # the 0..3 are LITTLE. + 'cpu_max_freq': { + '0..3': 1440000, + '4..5': 1824000 + }, + 'gpu_max_freq': 600000000, + }, + 'big_cores': ['4', '5'], + 'default_mode_governor': 'ondemand', }, - 'default_mode_governor': 'ondemand', - }, } + def _GetPerfModeDefinitions(product_model): if product_model.startswith('AOSP on '): product_model = product_model.replace('AOSP on ', '') return _PERFORMANCE_MODE_DEFINITIONS.get(product_model) + def _NoisyWarning(message): message += ' Results may be NOISY!!' logger.warning(message) @@ -144,7 +188,8 @@ class PerfControl(object): raw = self._ReadEachCpuFile(self._AVAILABLE_GOVERNORS_REL_PATH) self._available_governors = [ (cpu, raw_governors.strip().split() if not exit_code else None) - for cpu, raw_governors, exit_code in raw] + for cpu, raw_governors, exit_code in raw + ] def _SetMaxFrequenciesFromMode(self, mode): """Set maximum frequencies for GPU and CPU cores. @@ -165,8 +210,9 @@ class PerfControl(object): range_min, range_max = int(range_min), int(range_max) else: range_min = range_max = int(key) - cpu_files = ['cpu%d' % number - for number in xrange(range_min, range_max + 1)] + cpu_files = [ + 'cpu%d' % number for number in xrange(range_min, range_max + 1) + ] # Set the |max_frequency| on requested subset of the cores. self._SetScalingMaxFreqForCpus(max_frequency, ' '.join(cpu_files)) gpu_max_freq = mode.get('gpu_max_freq') @@ -197,6 +243,25 @@ class PerfControl(object): self.SetScalingGovernor('performance') self._SetMaxFrequenciesFromMode(high_perf_mode) + def SetLittleOnlyMode(self): + """Turns off big CPU cores on the device.""" + try: + self._device.EnableRoot() + except device_errors.CommandFailedError: + _NoisyWarning('Need root to turn off cores.') + return + mode_definitions = _GetPerfModeDefinitions(self._device.product_model) + if not mode_definitions: + _NoisyWarning('Unknown device: %s. Can\'t turn off cores.' + % self._device.product_model) + return + big_cores = mode_definitions.get('big_cores', []) + if not big_cores: + _NoisyWarning('No mode definition for device: %s.' % + self._device.product_model) + return + self._ForceCpusOffline(cpu_list=big_cores) + def SetDefaultPerfMode(self): """Sets the performance mode for the device to its default mode.""" if not self._device.HasRoot(): @@ -207,7 +272,7 @@ class PerfControl(object): else: default_mode_governor = mode_definitions.get('default_mode_governor') assert default_mode_governor, ('Default mode governor must be provided ' - 'for all perf mode definitions.') + 'for all perf mode definitions.') self.SetScalingGovernor(default_mode_governor) default_mode = mode_definitions.get('default_mode') if default_mode: @@ -226,9 +291,10 @@ class PerfControl(object): def GetCpuInfo(self): online = (output.rstrip() == '1' and status == 0 for (_, output, status) in self._ForEachCpu('cat "$CPU/online"')) - governor = (output.rstrip() if status == 0 else None - for (_, output, status) - in self._ForEachCpu('cat "$CPU/cpufreq/scaling_governor"')) + governor = ( + output.rstrip() if status == 0 else None + for (_, output, + status) in self._ForEachCpu('cat "$CPU/cpufreq/scaling_governor"')) return zip(self._cpu_files, online, governor) def _ForEachCpu(self, cmd, cpu_list=None): @@ -249,9 +315,7 @@ class PerfControl(object): cpu_list = self._cpu_file_list script = '; '.join([ 'for CPU in %s' % cpu_list, - 'do %s' % cmd, - 'echo -n "%~%$?%~%"', - 'done' + 'do %s' % cmd, 'echo -n "%~%$?%~%"', 'done' ]) output = self._device.RunShellCommand( script, cwd=self._CPU_PATH, check_return=True, as_root=True, shell=True) @@ -273,8 +337,7 @@ class PerfControl(object): self._ConditionallyWriteCpuFiles(path, value, cpu_files, condition='true') def _ReadEachCpuFile(self, path): - return self._ForEachCpu( - 'cat "$CPU/{path}"'.format(path=path)) + return self._ForEachCpu('cat "$CPU/{path}"'.format(path=path)) def SetScalingGovernor(self, value): """Sets the scaling governor to the given value on all possible CPUs. @@ -286,10 +349,9 @@ class PerfControl(object): value: [string] The new governor value. """ condition = 'test -e "{path}" && grep -q {value} {path}'.format( - path=('${CPU}/%s' % self._AVAILABLE_GOVERNORS_REL_PATH), - value=value) - self._ConditionallyWriteCpuFiles( - 'cpufreq/scaling_governor', value, self._cpu_file_list, condition) + path=('${CPU}/%s' % self._AVAILABLE_GOVERNORS_REL_PATH), value=value) + self._ConditionallyWriteCpuFiles('cpufreq/scaling_governor', value, + self._cpu_file_list, condition) def GetScalingGovernor(self): """Gets the currently set governor for each CPU. @@ -299,9 +361,8 @@ class PerfControl(object): governor. """ raw = self._ReadEachCpuFile('cpufreq/scaling_governor') - return [ - (cpu, raw_governor.strip() if not exit_code else None) - for cpu, raw_governor, exit_code in raw] + return [(cpu, raw_governor.strip() if not exit_code else None) + for cpu, raw_governor, exit_code in raw] def ListAvailableGovernors(self): """Returns the list of available governors for each CPU. @@ -316,9 +377,8 @@ class PerfControl(object): self._WriteCpuFiles('cpufreq/scaling_max_freq', '%d' % value, cpu_files) def _SetMaxGpuClock(self, value): - self._device.WriteFile('/sys/class/kgsl/kgsl-3d0/max_gpuclk', - str(value), - as_root=True) + self._device.WriteFile( + '/sys/class/kgsl/kgsl-3d0/max_gpuclk', str(value), as_root=True) def _AllCpusAreOnline(self): results = self._ForEachCpu('cat "$CPU/online"') @@ -326,8 +386,7 @@ class PerfControl(object): # is likely because on these devices it is impossible to bring the cpu0 # offline. Assuming the same for all devices until proven otherwise. return all(output.rstrip() == '1' and status == 0 - for (cpu, output, status) in results - if cpu != 'cpu0') + for (cpu, output, status) in results if cpu != 'cpu0') def _ForceAllCpusOnline(self, force_online): """Enable all CPUs on a device. @@ -352,3 +411,11 @@ class PerfControl(object): if force_online: self._ForEachCpu('echo 1 > "$CPU/online"') + + def _ForceCpusOffline(self, cpu_list): + """Disable selected CPUs on a device.""" + if self._have_mpdecision: + cmd = ['stop', 'mpdecision'] + self._device.RunShellCommand(cmd, check_return=True, as_root=True) + + self._ForEachCpu('echo 0 > "$CPU/online"', cpu_list=cpu_list) diff --git a/catapult/devil/devil/android/perf/perf_control_devicetest.py b/catapult/devil/devil/android/perf/perf_control_devicetest.py index b6458030..f73601a0 100644 --- a/catapult/devil/devil/android/perf/perf_control_devicetest.py +++ b/catapult/devil/devil/android/perf/perf_control_devicetest.py @@ -15,7 +15,6 @@ from devil.android.perf import perf_control class TestPerfControl(device_test_case.DeviceTestCase): - def setUp(self): super(TestPerfControl, self).setUp() if not os.getenv('BUILDTYPE'): @@ -34,5 +33,6 @@ class TestPerfControl(device_test_case.DeviceTestCase): finally: perf.SetDefaultPerfMode() + if __name__ == '__main__': unittest.main() diff --git a/catapult/devil/devil/android/perf/perf_control_test.py b/catapult/devil/devil/android/perf/perf_control_test.py index 3832424c..bde54d5a 100644 --- a/catapult/devil/devil/android/perf/perf_control_test.py +++ b/catapult/devil/devil/android/perf/perf_control_test.py @@ -14,9 +14,18 @@ with devil_env.SysPath(devil_env.PYMOCK_PATH): # pylint: disable=unused-argument -def _ShellCommandHandler(cmd, shell=False, check_return=False, - cwd=None, env=None, run_as=None, as_root=False, single_line=False, - large_output=False, raw_output=False, timeout=None, retries=None): +def _ShellCommandHandler(cmd, + shell=False, + check_return=False, + cwd=None, + env=None, + run_as=None, + as_root=False, + single_line=False, + large_output=False, + raw_output=False, + timeout=None, + retries=None): if cmd.startswith('for CPU in '): if 'scaling_available_governors' in cmd: contents = 'interactive ondemand userspace powersave performance' diff --git a/catapult/devil/devil/android/perf/surface_stats_collector.py b/catapult/devil/devil/android/perf/surface_stats_collector.py index f1140c12..6240624f 100644 --- a/catapult/devil/devil/android/perf/surface_stats_collector.py +++ b/catapult/devil/devil/android/perf/surface_stats_collector.py @@ -7,7 +7,6 @@ import Queue import re import threading - # Log marker containing SurfaceTexture timestamps. _SURFACE_TEXTURE_TIMESTAMPS_MESSAGE = 'SurfaceTexture update timestamps' _SURFACE_TEXTURE_TIMESTAMP_RE = r'\d+' @@ -75,8 +74,10 @@ class SurfaceStatsCollector(object): break raise Exception('Unable to get surface flinger latency data') - timestamps += [timestamp for timestamp in new_timestamps - if timestamp > last_timestamp] + timestamps += [ + timestamp for timestamp in new_timestamps + if timestamp > last_timestamp + ] if len(timestamps): last_timestamp = timestamps[-1] diff --git a/catapult/devil/devil/android/perf/surface_stats_collector_test.py b/catapult/devil/devil/android/perf/surface_stats_collector_test.py index 13b345ce..dda88ae0 100644 --- a/catapult/devil/devil/android/perf/surface_stats_collector_test.py +++ b/catapult/devil/devil/android/perf/surface_stats_collector_test.py @@ -14,7 +14,8 @@ class SurfaceStatsCollectorTests(unittest.TestCase): '7657467895508 7657482691352 7657493499756', '7657484466553 7657499645964 7657511077881', '7657500793457 7657516600576 7657527404785', - ], parse_timestamps=True) + ], + parse_timestamps=True) self.assertEqual( actual, (16.954612, [7657482.691352, 7657499.645964, 7657516.600576])) @@ -24,9 +25,9 @@ class SurfaceStatsCollectorTests(unittest.TestCase): '7657467895508 7657482691352 7657493499756', '7657484466553 7657499645964 7657511077881', '7657500793457 7657516600576 7657527404785', - ], parse_timestamps=False) - self.assertEqual( - actual, (16.954612, [])) + ], + parse_timestamps=False) + self.assertEqual(actual, (16.954612, [])) def testParseFrameData_withWarning(self): actual = surface_stats_collector.ParseFrameData([ @@ -35,6 +36,7 @@ class SurfaceStatsCollectorTests(unittest.TestCase): '7657467895508 7657482691352 7657493499756', '7657484466553 7657499645964 7657511077881', '7657500793457 7657516600576 7657527404785', - ], parse_timestamps=True) + ], + parse_timestamps=True) self.assertEqual( actual, (16.954612, [7657482.691352, 7657499.645964, 7657516.600576])) diff --git a/catapult/devil/devil/android/perf/thermal_throttle.py b/catapult/devil/devil/android/perf/thermal_throttle.py index fd6b08f3..9b8dc07b 100644 --- a/catapult/devil/devil/android/perf/thermal_throttle.py +++ b/catapult/devil/devil/android/perf/thermal_throttle.py @@ -39,6 +39,7 @@ class OmapThrottlingDetector(object): class ExynosThrottlingDetector(object): """Class to detect and track thermal throttling on an Exynos 5.""" + @staticmethod def IsSupported(device): return device.FileExists('/sys/bus/exynos5-core') @@ -98,8 +99,9 @@ class ThermalThrottle(object): return False has_been_throttled = False serial_number = str(self._device) - log = self._device.RunShellCommand( - ['dmesg', '-c'], large_output=True, check_return=True) + log = self._device.RunShellCommand(['dmesg', '-c'], + large_output=True, + check_return=True) degree_symbol = unichr(0x00B0) for line in log: if self._detector.BecameThrottled(line): @@ -114,19 +116,19 @@ class ThermalThrottle(object): has_been_throttled = True temperature = self._detector.GetThrottlingTemperature(line) if temperature is not None: - logger.info(u'Device %s thermally throttled at %3.1f%sC', - serial_number, temperature, degree_symbol) + logger.info(u'Device %s thermally throttled at %3.1f%sC', serial_number, + temperature, degree_symbol) if logger.isEnabledFor(logging.DEBUG): # Print current temperature of CPU SoC. temperature = self._detector.GetCurrentTemperature() if temperature is not None: - logger.debug(u'Current SoC temperature of %s = %3.1f%sC', - serial_number, temperature, degree_symbol) + logger.debug(u'Current SoC temperature of %s = %3.1f%sC', serial_number, + temperature, degree_symbol) # Print temperature of battery, to give a system temperature - dumpsys_log = self._device.RunShellCommand( - ['dumpsys', 'battery'], check_return=True) + dumpsys_log = self._device.RunShellCommand(['dumpsys', 'battery'], + check_return=True) for line in dumpsys_log: if 'temperature' in line: btemp = float([s for s in line.split() if s.isdigit()][0]) / 10.0 diff --git a/catapult/devil/devil/android/ports.py b/catapult/devil/devil/android/ports.py index 4547a627..f25c8bfd 100644 --- a/catapult/devil/devil/android/ports.py +++ b/catapult/devil/devil/android/ports.py @@ -1,7 +1,6 @@ # Copyright (c) 2012 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Functions that deal with local and device ports.""" import contextlib @@ -63,8 +62,7 @@ def AllocateTestServerPort(): while not IsHostPortAvailable(port): port += 1 ports_tried.append(port) - if (port > _TEST_SERVER_PORT_LAST or - port < _TEST_SERVER_PORT_FIRST): + if port > _TEST_SERVER_PORT_LAST or port < _TEST_SERVER_PORT_FIRST: port = 0 else: fp.seek(0, os.SEEK_SET) @@ -78,8 +76,9 @@ def AllocateTestServerPort(): if port: logger.info('Allocate port %d for test server.', port) else: - logger.error('Could not allocate port for test server. ' - 'List of ports tried: %s', str(ports_tried)) + logger.error( + 'Could not allocate port for test server. ' + 'List of ports tried: %s', str(ports_tried)) return port @@ -115,8 +114,9 @@ def IsDevicePortUsed(device, device_port, state=''): True if the port on device is already used, otherwise returns False. """ base_urls = ('127.0.0.1:%d' % device_port, 'localhost:%d' % device_port) - netstat_results = device.RunShellCommand( - ['netstat', '-an'], check_return=True, large_output=True) + netstat_results = device.RunShellCommand(['netstat', '-an'], + check_return=True, + large_output=True) for single_connect in netstat_results: # Column 3 is the local address which we want to check with. connect_results = single_connect.split() @@ -131,8 +131,13 @@ def IsDevicePortUsed(device, device_port, state=''): return False -def IsHttpServerConnectable(host, port, tries=3, command='GET', path='/', - expected_read='', timeout=2): +def IsHttpServerConnectable(host, + port, + tries=3, + command='GET', + path='/', + expected_read='', + timeout=2): """Checks whether the specified http server is ready to serve request or not. Args: @@ -157,8 +162,8 @@ def IsHttpServerConnectable(host, port, tries=3, command='GET', path='/', for i in xrange(0, tries): client_error = None try: - with contextlib.closing(httplib.HTTPConnection( - host, port, timeout=timeout)) as http: + with contextlib.closing( + httplib.HTTPConnection(host, port, timeout=timeout)) as http: # Output some debug information when we have tried more than 2 times. http.set_debuglevel(i >= 2) http.request(command, path) @@ -167,8 +172,8 @@ def IsHttpServerConnectable(host, port, tries=3, command='GET', path='/', if r.status == 200 and r.reason == 'OK' and content == expected_read: return (True, '') client_error = ('Bad response: %s %s version %s\n ' % - (r.status, r.reason, r.version) + - '\n '.join([': '.join(h) for h in r.getheaders()])) + (r.status, r.reason, r.version) + '\n '.join( + [': '.join(h) for h in r.getheaders()])) except (httplib.HTTPException, socket.error) as e: # Probably too quick connecting: try again. exception_error_msgs = traceback.format_exception_only(type(e), e) diff --git a/catapult/devil/devil/android/sdk/aapt.py b/catapult/devil/devil/android/sdk/aapt.py index 7ae3a938..76c7ef68 100644 --- a/catapult/devil/devil/android/sdk/aapt.py +++ b/catapult/devil/devil/android/sdk/aapt.py @@ -1,14 +1,12 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """This module wraps the Android Asset Packaging Tool.""" from devil.android.sdk import build_tools from devil.utils import cmd_helper from devil.utils import lazy - _aapt_path = lazy.WeakConstant(lambda: build_tools.GetPath('aapt')) diff --git a/catapult/devil/devil/android/sdk/adb_compatibility_devicetest.py b/catapult/devil/devil/android/sdk/adb_compatibility_devicetest.py index cbe2a1b6..d2c2d4f5 100644 --- a/catapult/devil/devil/android/sdk/adb_compatibility_devicetest.py +++ b/catapult/devil/devil/android/sdk/adb_compatibility_devicetest.py @@ -11,8 +11,8 @@ import signal import sys import unittest -_CATAPULT_BASE_DIR = os.path.abspath(os.path.join( - os.path.dirname(__file__), '..', '..', '..', '..')) +_CATAPULT_BASE_DIR = os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..', '..')) sys.path.append(os.path.join(_CATAPULT_BASE_DIR, 'devil')) from devil import devil_env @@ -22,9 +22,8 @@ from devil.android.sdk import adb_wrapper from devil.utils import cmd_helper from devil.utils import timeout_retry - -_TEST_DATA_DIR = os.path.abspath(os.path.join( - os.path.dirname(__file__), 'test', 'data')) +_TEST_DATA_DIR = os.path.abspath( + os.path.join(os.path.dirname(__file__), 'test', 'data')) def _hostAdbPids(): @@ -34,26 +33,24 @@ def _hostAdbPids(): return [] pids_and_names = (line.split() for line in ps_output.splitlines()) - return [int(pid) for pid, name in pids_and_names - if name == 'adb'] + return [int(pid) for pid, name in pids_and_names if name == 'adb'] class AdbCompatibilityTest(device_test_case.DeviceTestCase): - @classmethod def setUpClass(cls): custom_adb_path = os.environ.get('ADB_PATH') custom_deps = { - 'config_type': 'BaseConfig', - 'dependencies': {}, + 'config_type': 'BaseConfig', + 'dependencies': {}, } if custom_adb_path: custom_deps['dependencies']['adb'] = { - 'file_info': { - devil_env.GetPlatform(): { - 'local_paths': [custom_adb_path], + 'file_info': { + devil_env.GetPlatform(): { + 'local_paths': [custom_adb_path], + }, }, - }, } devil_env.config.Initialize(configs=[custom_deps]) @@ -124,7 +121,7 @@ class AdbCompatibilityTest(device_test_case.DeviceTestCase): if not external_storage: self.skipTest('External storage not available.') while True: - random_hex = hex(random.randint(0, 2 ** 52))[2:] + random_hex = hex(random.randint(0, 2**52))[2:] name = 'tmp_push_test%s' % random_hex path = posixpath.join(external_storage, name) try: @@ -156,9 +153,8 @@ class AdbCompatibilityTest(device_test_case.DeviceTestCase): with self.assertRaises(device_errors.AdbShellCommandFailedError): under_test.Shell('ls %s' % resulting_file) under_test.Push(src, dest) - self.assertEquals( - resulting_file, - under_test.Shell('ls %s' % resulting_file).strip()) + self.assertEquals(resulting_file, + under_test.Shell('ls %s' % resulting_file).strip()) def testPush_directoryToDirectory(self): under_test = self.getTestInstance() diff --git a/catapult/devil/devil/android/sdk/adb_wrapper.py b/catapult/devil/devil/android/sdk/adb_wrapper.py index 13c0f520..8cd73136 100644 --- a/catapult/devil/devil/android/sdk/adb_wrapper.py +++ b/catapult/devil/devil/android/sdk/adb_wrapper.py @@ -1,7 +1,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """This module wraps Android's adb tool. This is a thin wrapper around the adb interface. Any additional complexity @@ -32,7 +31,6 @@ with devil_env.SysPath(devil_env.DEPENDENCY_MANAGER_PATH): logger = logging.getLogger(__name__) - ADB_KEYS_FILE = '/data/misc/adb/adb_keys' ADB_HOST_KEYS_DIR = os.path.join(os.path.expanduser('~'), '.android') @@ -76,8 +74,8 @@ def _FindAdb(): pass try: - return os.path.join(devil_env.config.LocalPath('android_sdk'), - 'platform-tools', 'adb') + return os.path.join( + devil_env.config.LocalPath('android_sdk'), 'platform-tools', 'adb') except dependency_manager.NoPathFoundError: pass @@ -101,8 +99,8 @@ def _ShouldRetryAdbCmd(exc): # Errors are potentially transient and should be retried, with the exception # of NoAdbError. Exceptions [e.g. generated from SIGTERM handler] should be # raised. - return (isinstance(exc, base_error.BaseError) and - not isinstance(exc, device_errors.NoAdbError)) + return (isinstance(exc, base_error.BaseError) + and not isinstance(exc, device_errors.NoAdbError)) DeviceStat = collections.namedtuple('DeviceStat', @@ -157,6 +155,7 @@ class AdbWrapper(object): pshell.RunCommand('which ls') pshell.RunCommand('echo TEST', close=True) ''' + def __init__(self, serial): """Initialization function: @@ -179,11 +178,12 @@ class AdbWrapper(object): if self._process is not None: raise RuntimeError('Persistent shell already running.') # pylint: disable=protected-access - self._process = subprocess.Popen(self._cmd, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - shell=False, - env=AdbWrapper._ADB_ENV) + self._process = subprocess.Popen( + self._cmd, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + shell=False, + env=AdbWrapper._ADB_ENV) def WaitForReady(self): """Wait for the shell to be ready after starting. @@ -210,6 +210,7 @@ class AdbWrapper(object): """ if close: + def run_cmd(cmd): send_cmd = '( %s ); echo $?; exit;\n' % cmd.rstrip() (output, _) = self._process.communicate(send_cmd) @@ -218,6 +219,7 @@ class AdbWrapper(object): yield x else: + def run_cmd(cmd): send_cmd = '( %s ); echo DONE:$?;\n' % cmd.rstrip() self._process.stdin.write(send_cmd) @@ -228,8 +230,10 @@ class AdbWrapper(object): break yield output_line - result = [line for line in run_cmd(command) - if not _IsExtraneousLine(line, command)] + result = [ + line for line in run_cmd(command) + if not _IsExtraneousLine(line, command) + ] return (result[:-1], int(result[-1])) @@ -262,8 +266,14 @@ class AdbWrapper(object): # pylint: disable=unused-argument @classmethod @decorators.WithTimeoutAndConditionalRetries(_ShouldRetryAdbCmd) - def _RunAdbCmd(cls, args, timeout=None, retries=None, device_serial=None, - check_error=True, cpu_affinity=None, additional_env=None): + def _RunAdbCmd(cls, + args, + timeout=None, + retries=None, + device_serial=None, + check_error=True, + cpu_affinity=None, + additional_env=None): if timeout: remaining = timeout_retry.CurrentTimeoutThreadGroup().GetRemainingTime() if remaining: @@ -276,14 +286,18 @@ class AdbWrapper(object): if additional_env: env.update(additional_env) try: - status, output = cmd_helper.GetCmdStatusAndOutputWithTimeout( - cls._BuildAdbCmd(args, device_serial, cpu_affinity=cpu_affinity), - timeout, env=env) + adb_cmd = cls._BuildAdbCmd(args, device_serial, cpu_affinity=cpu_affinity) + status, output = cmd_helper.GetCmdStatusAndOutputWithTimeout(adb_cmd, + timeout, + env=env) except OSError as e: if e.errno in (errno.ENOENT, errno.ENOEXEC): raise device_errors.NoAdbError(msg=str(e)) else: raise + except cmd_helper.TimeoutError: + logger.error('Timeout on adb command: %r', adb_cmd) + raise # Best effort to catch errors from adb; unfortunately adb is very # inconsistent with error reporting so many command failures present @@ -292,14 +306,15 @@ class AdbWrapper(object): not_found_m = _DEVICE_NOT_FOUND_RE.match(output) device_waiting_m = _WAITING_FOR_DEVICE_RE.match(output) if (device_waiting_m is not None - or (not_found_m is not None and - not_found_m.group('serial') == device_serial)): + or (not_found_m is not None + and not_found_m.group('serial') == device_serial)): raise device_errors.DeviceUnreachableError(device_serial) else: - raise device_errors.AdbCommandFailedError( - args, output, status, device_serial) + raise device_errors.AdbCommandFailedError(args, output, status, + device_serial) return output + # pylint: enable=unused-argument def _RunDeviceAdbCmd(self, args, timeout, retries, check_error=True): @@ -316,12 +331,14 @@ class AdbWrapper(object): Returns: The output of the command. """ - return self._RunAdbCmd(args, timeout=timeout, retries=retries, - device_serial=self._device_serial, - check_error=check_error) + return self._RunAdbCmd( + args, + timeout=timeout, + retries=retries, + device_serial=self._device_serial, + check_error=check_error) - def _IterRunDeviceAdbCmd(self, args, iter_timeout, timeout, - check_error=True): + def _IterRunDeviceAdbCmd(self, args, iter_timeout, timeout, check_error=True): """Runs an adb command and returns an iterator over its output lines. Args: @@ -371,6 +388,7 @@ class AdbWrapper(object): output = [int(x) for x in output.split()] logger.info('PIDs for adb found: %r', output) return status == 0 + # pylint: enable=unused-argument @classmethod @@ -378,7 +396,9 @@ class AdbWrapper(object): cls._RunAdbCmd(['kill-server'], timeout=timeout, retries=retries) @classmethod - def StartServer(cls, keys=None, timeout=DEFAULT_TIMEOUT, + def StartServer(cls, + keys=None, + timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): """Starts the ADB server. @@ -391,8 +411,11 @@ class AdbWrapper(object): if keys: additional_env['ADB_VENDOR_KEYS'] = ':'.join(keys) # CPU affinity is used to reduce adb instability http://crbug.com/268450 - cls._RunAdbCmd(['start-server'], timeout=timeout, retries=retries, - cpu_affinity=0, additional_env=additional_env) + cls._RunAdbCmd(['start-server'], + timeout=timeout, + retries=retries, + cpu_affinity=0, + additional_env=additional_env) @classmethod def GetDevices(cls, timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): @@ -401,8 +424,11 @@ class AdbWrapper(object): return cls.Devices(timeout=timeout, retries=retries) @classmethod - def Devices(cls, desired_state=_READY_STATE, long_list=False, - timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): + def Devices(cls, + desired_state=_READY_STATE, + long_list=False, + timeout=DEFAULT_TIMEOUT, + retries=DEFAULT_RETRIES): """Get the list of active attached devices. Args: @@ -415,23 +441,22 @@ class AdbWrapper(object): Yields: AdbWrapper instances. """ - lines = cls._RawDevices(long_list=long_list, timeout=timeout, - retries=retries) + lines = cls._RawDevices( + long_list=long_list, timeout=timeout, retries=retries) if long_list: - return [ - [AdbWrapper(line[0])] + line[1:] - for line in lines - if (len(line) >= 2 and (not desired_state or line[1] == desired_state)) - ] + return [[AdbWrapper(line[0])] + line[1:] for line in lines if ( + len(line) >= 2 and (not desired_state or line[1] == desired_state))] else: return [ - AdbWrapper(line[0]) - for line in lines - if (len(line) == 2 and (not desired_state or line[1] == desired_state)) + AdbWrapper(line[0]) for line in lines + if (len(line) == 2 and (not desired_state or line[1] == desired_state) + ) ] @classmethod - def _RawDevices(cls, long_list=False, timeout=DEFAULT_TIMEOUT, + def _RawDevices(cls, + long_list=False, + timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): cmd = ['devices'] if long_list: @@ -447,14 +472,24 @@ class AdbWrapper(object): """ return self._device_serial - def Push(self, local, remote, timeout=60 * 5, retries=DEFAULT_RETRIES): + def Push(self, + local, + remote, + sync=False, + timeout=60 * 5, + retries=DEFAULT_RETRIES): """Pushes a file from the host to the device. Args: local: Path on the host filesystem. remote: Path on the device filesystem. + sync: (optional) Whether to only push files that are newer on the host. + Not supported when using adb prior to 1.0.39. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. + + Raises: + AdbVersionError if sync=True with versions of adb prior to 1.0.39. """ VerifyLocalFileExists(local) @@ -495,7 +530,24 @@ class AdbWrapper(object): # without modification. pass - self._RunDeviceAdbCmd(['push', local, remote], timeout, retries) + push_cmd = ['push'] + + if sync: + push_cmd += ['--sync'] + if (du_version.LooseVersion(self.Version()) < + du_version.LooseVersion('1.0.39')): + # The --sync flag for `adb push` is a relatively recent addition. + # We're not sure exactly which release first contained it, but it + # exists at least as far back as 1.0.39. + raise device_errors.AdbVersionError( + push_cmd, + desc='--sync not supported', + actual_version=self.Version(), + min_version='1.0.39') + + push_cmd += [local, remote] + + self._RunDeviceAdbCmd(push_cmd, timeout, retries) def Pull(self, remote, local, timeout=60 * 5, retries=DEFAULT_RETRIES): """Pulls a file from the device to the host. @@ -529,7 +581,10 @@ class AdbWrapper(object): return cmd_helper.StartCmd( self._BuildAdbCmd(['shell'] + cmd, self._device_serial)) - def Shell(self, command, expect_status=0, timeout=DEFAULT_TIMEOUT, + def Shell(self, + command, + expect_status=0, + timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): """Runs a shell command on the device. @@ -582,8 +637,9 @@ class AdbWrapper(object): """ args = ['shell', command] return cmd_helper.IterCmdOutputLines( - self._BuildAdbCmd(args, self._device_serial), timeout=timeout, - env=self._ADB_ENV) + self._BuildAdbCmd(args, self._device_serial), + timeout=timeout, + env=self._ADB_ENV) def Ls(self, path, timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): """List the contents of a directory on the device. @@ -602,12 +658,16 @@ class AdbWrapper(object): directory in the device, or the output of "adb ls" command is less than four columns """ + def ParseLine(line, cmd): cols = line.split(None, 3) if len(cols) < 4: raise device_errors.AdbCommandFailedError( - cmd, line, "the output should be 4 columns, but is only %d columns" - % len(cols), device_serial=self._device_serial) + cmd, + line, + "the output should be 4 columns, but is only %d columns" % + len(cols), + device_serial=self._device_serial) filename = cols.pop() stat = DeviceStat(*[int(num, base=16) for num in cols]) return (filename, stat) @@ -619,12 +679,20 @@ class AdbWrapper(object): return [ParseLine(line, cmd) for line in lines] else: raise device_errors.AdbCommandFailedError( - cmd, 'path does not specify an accessible directory in the device', + cmd, + 'path does not specify an accessible directory in the device', device_serial=self._device_serial) - def Logcat(self, clear=False, dump=False, filter_specs=None, - logcat_format=None, ring_buffer=None, iter_timeout=None, - check_error=True, timeout=None, retries=DEFAULT_RETRIES): + def Logcat(self, + clear=False, + dump=False, + filter_specs=None, + logcat_format=None, + ring_buffer=None, + iter_timeout=None, + check_error=True, + timeout=None, + retries=DEFAULT_RETRIES): """Get an iterable over the logcat output. Args: @@ -674,8 +742,12 @@ class AdbWrapper(object): cmd, timeout, retries, check_error=check_error) return output.splitlines() - def Forward(self, local, remote, allow_rebind=False, - timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): + def Forward(self, + local, + remote, + allow_rebind=False, + timeout=DEFAULT_TIMEOUT, + retries=DEFAULT_RETRIES): """Forward socket connections from the local socket to the remote socket. Sockets are specified by one of: @@ -703,7 +775,9 @@ class AdbWrapper(object): if output: logger.warning('Unexpected output from "adb forward": %s', output) - def ForwardRemove(self, local, timeout=DEFAULT_TIMEOUT, + def ForwardRemove(self, + local, + timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): """Remove a forward socket connection. @@ -712,8 +786,7 @@ class AdbWrapper(object): timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. """ - self._RunDeviceAdbCmd(['forward', '--remove', str(local)], timeout, - retries) + self._RunDeviceAdbCmd(['forward', '--remove', str(local)], timeout, retries) def ForwardList(self, timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): """List all currently forwarded socket connections. @@ -747,11 +820,19 @@ class AdbWrapper(object): Returns: A list of PIDs as strings. """ - return [a.strip() for a in - self._RunDeviceAdbCmd(['jdwp'], timeout, retries).split('\n')] - - def Install(self, apk_path, forward_lock=False, allow_downgrade=False, - reinstall=False, sd_card=False, timeout=60 * 2, + return [ + a.strip() + for a in self._RunDeviceAdbCmd(['jdwp'], timeout, retries).split('\n') + ] + + def Install(self, + apk_path, + forward_lock=False, + allow_downgrade=False, + reinstall=False, + sd_card=False, + streaming=None, + timeout=60 * 2, retries=DEFAULT_RETRIES): """Install an apk on the device. @@ -761,6 +842,10 @@ class AdbWrapper(object): allow_downgrade: (optional) If set, allows for downgrades. reinstall: (optional) If set reinstalls the app, keeping its data. sd_card: (optional) If set installs on the SD card. + streaming: (optional) If not set, use default way to install. + If True, performs streaming install. + If False, app is pushed to device and be installed from there. + Note this option is not supported prior to adb version 1.0.40 timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. """ @@ -774,15 +859,32 @@ class AdbWrapper(object): cmd.append('-s') if allow_downgrade: cmd.append('-d') + if streaming in (True, False): + if (du_version.LooseVersion(self.Version()) < + du_version.LooseVersion('1.0.40')): + logging.warning( + 'adb: streaming options not supported prior to version 1.0.40 ' + '(current: %s)', self.Version()) + elif streaming: + cmd.append('--streaming') + else: + cmd.append('--no-streaming') cmd.append(apk_path) output = self._RunDeviceAdbCmd(cmd, timeout, retries) if 'Success' not in output: raise device_errors.AdbCommandFailedError( cmd, output, device_serial=self._device_serial) - def InstallMultiple(self, apk_paths, forward_lock=False, reinstall=False, - sd_card=False, allow_downgrade=False, partial=False, - timeout=60 * 2, retries=DEFAULT_RETRIES): + def InstallMultiple(self, + apk_paths, + forward_lock=False, + reinstall=False, + sd_card=False, + allow_downgrade=False, + partial=False, + streaming=None, + timeout=60 * 2, + retries=DEFAULT_RETRIES): """Install an apk with splits on the device. Args: @@ -792,6 +894,10 @@ class AdbWrapper(object): sd_card: (optional) If set installs on the SD card. allow_downgrade: (optional) Allow versionCode downgrade. partial: (optional) Package ID if apk_paths doesn't include all .apks. + streaming: (optional) If not set, use default way to install. + If True, performs streaming install. + If False, app is pushed to device and be installed from there. + Note this option is not supported prior to adb version 1.0.40 timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. """ @@ -806,6 +912,16 @@ class AdbWrapper(object): cmd.append('-s') if allow_downgrade: cmd.append('-d') + if streaming in (True, False): + if (du_version.LooseVersion(self.Version()) < + du_version.LooseVersion('1.0.40')): + logging.warning( + 'adb: streaming options not supported prior to version 1.0.40 ' + '(current: %s)', self.Version()) + elif streaming: + cmd.append('--streaming') + else: + cmd.append('--no-streaming') if partial: cmd.extend(('-p', partial)) cmd.extend(apk_paths) @@ -814,7 +930,10 @@ class AdbWrapper(object): raise device_errors.AdbCommandFailedError( cmd, output, device_serial=self._device_serial) - def Uninstall(self, package, keep_data=False, timeout=DEFAULT_TIMEOUT, + def Uninstall(self, + package, + keep_data=False, + timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): """Remove the app |package| from the device. @@ -833,8 +952,14 @@ class AdbWrapper(object): raise device_errors.AdbCommandFailedError( cmd, output, device_serial=self._device_serial) - def Backup(self, path, packages=None, apk=False, shared=False, - nosystem=True, include_all=False, timeout=DEFAULT_TIMEOUT, + def Backup(self, + path, + packages=None, + apk=False, + shared=False, + nosystem=True, + include_all=False, + timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): """Write an archive of the device's data to |path|. @@ -949,8 +1074,7 @@ class AdbWrapper(object): raise device_errors.AdbCommandFailedError( ['root'], output, device_serial=self._device_serial) - def Emu(self, cmd, timeout=DEFAULT_TIMEOUT, - retries=DEFAULT_RETRIES): + def Emu(self, cmd, timeout=DEFAULT_TIMEOUT, retries=DEFAULT_RETRIES): """Runs an emulator console command. See http://developer.android.com/tools/devices/emulator.html#console diff --git a/catapult/devil/devil/android/sdk/adb_wrapper_devicetest.py b/catapult/devil/devil/android/sdk/adb_wrapper_devicetest.py index b0ccb24a..f8b010b8 100755 --- a/catapult/devil/devil/android/sdk/adb_wrapper_devicetest.py +++ b/catapult/devil/devil/android/sdk/adb_wrapper_devicetest.py @@ -3,7 +3,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Tests for the AdbWrapper class.""" import os @@ -17,7 +16,6 @@ from devil.android.sdk import adb_wrapper class TestAdbWrapper(device_test_case.DeviceTestCase): - def setUp(self): super(TestAdbWrapper, self).setUp() self._adb = adb_wrapper.AdbWrapper(self.serial) @@ -54,7 +52,7 @@ class TestAdbWrapper(device_test_case.DeviceTestCase): def testPersistentShell(self): # We need to access the device serial number here in order # to create the persistent shell. - serial = self._adb.GetDeviceSerial() # pylint: disable=protected-access + serial = self._adb.GetDeviceSerial() # pylint: disable=protected-access with self._adb.PersistentShell(serial) as pshell: (res1, code1) = pshell.RunCommand('echo TEST') (res2, code2) = pshell.RunCommand('echo TEST2') diff --git a/catapult/devil/devil/android/sdk/adb_wrapper_test.py b/catapult/devil/devil/android/sdk/adb_wrapper_test.py index 07f784d0..f30ab152 100755 --- a/catapult/devil/devil/android/sdk/adb_wrapper_test.py +++ b/catapult/devil/devil/android/sdk/adb_wrapper_test.py @@ -2,7 +2,6 @@ # Copyright 2017 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Unit tests for some APIs with conditional logic in adb_wrapper.py """ @@ -24,8 +23,7 @@ class AdbWrapperTest(unittest.TestCase): def _MockRunDeviceAdbCmd(self, return_value): return mock.patch.object( - self.adb, - '_RunDeviceAdbCmd', + self.adb, '_RunDeviceAdbCmd', mock.Mock(side_effect=None, return_value=return_value)) def testDisableVerityWhenDisabled(self): @@ -50,23 +48,23 @@ class AdbWrapperTest(unittest.TestCase): def testFailEnableVerity(self): with self._MockRunDeviceAdbCmd('error: closed'): - self.assertRaises( - device_errors.AdbCommandFailedError, self.adb.EnableVerity) + self.assertRaises(device_errors.AdbCommandFailedError, + self.adb.EnableVerity) def testFailDisableVerity(self): with self._MockRunDeviceAdbCmd('error: closed'): - self.assertRaises( - device_errors.AdbCommandFailedError, self.adb.DisableVerity) + self.assertRaises(device_errors.AdbCommandFailedError, + self.adb.DisableVerity) @mock.patch('devil.utils.cmd_helper.GetCmdStatusAndOutputWithTimeout') def testDeviceUnreachable(self, get_cmd_mock): get_cmd_mock.return_value = ( 1, "error: device '%s' not found" % self.device_serial) - self.assertRaises( - device_errors.DeviceUnreachableError, self.adb.Shell, '/bin/true') + self.assertRaises(device_errors.DeviceUnreachableError, self.adb.Shell, + '/bin/true') @mock.patch('devil.utils.cmd_helper.GetCmdStatusAndOutputWithTimeout') def testWaitingForDevice(self, get_cmd_mock): get_cmd_mock.return_value = (1, '- waiting for device - ') - self.assertRaises( - device_errors.DeviceUnreachableError, self.adb.Shell, '/bin/true') + self.assertRaises(device_errors.DeviceUnreachableError, self.adb.Shell, + '/bin/true') diff --git a/catapult/devil/devil/android/sdk/build_tools.py b/catapult/devil/devil/android/sdk/build_tools.py index 99083d99..64cc96d1 100644 --- a/catapult/devil/devil/android/sdk/build_tools.py +++ b/catapult/devil/devil/android/sdk/build_tools.py @@ -27,8 +27,16 @@ def GetPath(build_tool): def _PathInLocalSdk(build_tool): build_tools_path = _build_tools_path.read() - return (os.path.join(build_tools_path, build_tool) if build_tools_path - else None) + if not build_tools_path: + raise dependency_manager.NoPathFoundError(build_tool, + devil_env.GetPlatform()) + + candidate_path = os.path.join(build_tools_path, build_tool) + if not os.path.exists(candidate_path): + raise dependency_manager.NoPathFoundError(build_tool, + devil_env.GetPlatform()) + + return candidate_path def _FindBuildTools(): diff --git a/catapult/devil/devil/android/sdk/bundletool.py b/catapult/devil/devil/android/sdk/bundletool.py new file mode 100644 index 00000000..5c181c56 --- /dev/null +++ b/catapult/devil/devil/android/sdk/bundletool.py @@ -0,0 +1,62 @@ +# Copyright 2019 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""This module wraps bundletool.""" + +import json + +from devil import base_error +from devil import devil_env +from devil.utils import cmd_helper +from devil.utils import lazy +from py_utils import tempfile_ext + +_bundletool_path = lazy.WeakConstant(lambda: devil_env.config.FetchPath( + 'bundletool')) + + +def ExtractApks(output_dir, + apks_path, + abis, + locales, + features, + pixel_density, + sdk_version, + modules=None): + """Extracts splits from APKS archive. + + Args: + output_dir: Directory to extract splits into. + apks_path: Path to APKS archive. + abis: ABIs to support. + locales: Locales to support. + features: Device features to support. + pixel_density: Pixel density to support. + sdk_version: Target SDK version. + modules: Extra modules to install. + """ + device_spec = { + 'supportedAbis': abis, + 'supportedLocales': ['%s-%s' % l for l in locales], + 'deviceFeatures': features, + 'screenDensity': pixel_density, + 'sdkVersion': sdk_version, + } + with tempfile_ext.TemporaryFileName(suffix='.json') as device_spec_path: + with open(device_spec_path, 'w') as f: + json.dump(device_spec, f) + cmd = [ + 'java', + '-jar', + _bundletool_path.read(), + 'extract-apks', + '--apks=%s' % apks_path, + '--device-spec=%s' % device_spec_path, + '--output-dir=%s' % output_dir, + ] + if modules: + cmd += ['--modules=%s' % ','.join(modules)] + status, stdout, stderr = cmd_helper.GetCmdStatusOutputAndError(cmd) + if status != 0: + raise base_error.BaseError('Failed running {} with output\n{}\n{}'.format( + ' '.join(cmd), stdout, stderr)) diff --git a/catapult/devil/devil/android/sdk/dexdump.py b/catapult/devil/devil/android/sdk/dexdump.py index 992366e8..2a59e9bf 100644 --- a/catapult/devil/devil/android/sdk/dexdump.py +++ b/catapult/devil/devil/android/sdk/dexdump.py @@ -6,7 +6,6 @@ from devil.android.sdk import build_tools from devil.utils import cmd_helper from devil.utils import lazy - _dexdump_path = lazy.WeakConstant(lambda: build_tools.GetPath('dexdump')) @@ -28,4 +27,3 @@ def DexDump(dexfiles, file_summary=False): args.append('-f') return cmd_helper.IterCmdOutputLines(args) - diff --git a/catapult/devil/devil/android/sdk/fastboot.py b/catapult/devil/devil/android/sdk/fastboot.py index 47f4167f..d0f85667 100644 --- a/catapult/devil/devil/android/sdk/fastboot.py +++ b/catapult/devil/devil/android/sdk/fastboot.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """This module wraps Android's fastboot tool. This is a thin wrapper around the fastboot interface. Any additional complexity @@ -22,10 +21,12 @@ _FLASH_TIMEOUT = _DEFAULT_TIMEOUT * 10 class Fastboot(object): - _fastboot_path = lazy.WeakConstant( - lambda: devil_env.config.FetchPath('fastboot')) + _fastboot_path = lazy.WeakConstant(lambda: devil_env.config.FetchPath( + 'fastboot')) - def __init__(self, device_serial, default_timeout=_DEFAULT_TIMEOUT, + def __init__(self, + device_serial, + default_timeout=_DEFAULT_TIMEOUT, default_retries=_DEFAULT_RETRIES): """Initializes the FastbootWrapper. @@ -57,9 +58,10 @@ class Fastboot(object): if isinstance(cmd, list): cmd = [cls._fastboot_path.read()] + cmd else: - raise TypeError( - 'Command for _RunDeviceFastbootCommand must be a list.') - status, output = cmd_helper.GetCmdStatusAndOutput(cmd) + raise TypeError('Command for _RunDeviceFastbootCommand must be a list.') + # fastboot can't be trusted to keep non-error output out of stderr, so + # capture stderr as part of stdout. + status, output = cmd_helper.GetCmdStatusAndOutput(cmd, merge_stderr=True) if int(status) != 0: raise device_errors.FastbootCommandFailedError(cmd, output, status) return output @@ -80,6 +82,19 @@ class Fastboot(object): cmd = ['-s', self._device_serial] + cmd return self._RunFastbootCommand(cmd) + @decorators.WithTimeoutAndRetriesDefaults(_DEFAULT_TIMEOUT, _DEFAULT_RETRIES) + def GetVar(self, variable, timeout=None, retries=None): + args = ['getvar', variable] + output = self._RunDeviceFastbootCommand(args) + # getvar returns timing information on the last line of output, so only + # parse the first line. + output = output.splitlines()[0] + # And the first line should match the format '$(var): $(value)'. + if variable + ': ' not in output: + raise device_errors.FastbootCommandFailedError( + args, output, message="Unknown 'getvar' output format.") + return output.split('%s: ' % variable)[1].strip() + @decorators.WithTimeoutAndRetriesDefaults(_FLASH_TIMEOUT, 0) def Flash(self, partition, image, timeout=None, retries=None): """Flash partition with img. @@ -118,5 +133,4 @@ class Fastboot(object): Args: value: boolean value to set off-mode-charging on or off. """ - self._RunDeviceFastbootCommand( - ['oem', 'off-mode-charge', str(int(value))]) + self._RunDeviceFastbootCommand(['oem', 'off-mode-charge', str(int(value))]) diff --git a/catapult/devil/devil/android/sdk/gce_adb_wrapper.py b/catapult/devil/devil/android/sdk/gce_adb_wrapper.py index 71600f40..c6061973 100644 --- a/catapult/devil/devil/android/sdk/gce_adb_wrapper.py +++ b/catapult/devil/devil/android/sdk/gce_adb_wrapper.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Provides a work around for various adb commands on android gce instances. Some adb commands don't work well when the device is a cloud vm, namely @@ -22,14 +21,14 @@ logger = logging.getLogger(__name__) class GceAdbWrapper(adb_wrapper.AdbWrapper): - def __init__(self, device_serial): super(GceAdbWrapper, self).__init__(device_serial) self._Connect() self.Root() self._instance_ip = self.Shell('getprop net.gce.ip').strip() - def _Connect(self, timeout=adb_wrapper.DEFAULT_TIMEOUT, + def _Connect(self, + timeout=adb_wrapper.DEFAULT_TIMEOUT, retries=adb_wrapper.DEFAULT_RETRIES): """Connects ADB to the android gce instance.""" cmd = ['connect', self._device_serial] @@ -59,8 +58,8 @@ class GceAdbWrapper(adb_wrapper.AdbWrapper): # with the destination dir. So if local is a dir, just scp its contents. for f in os.listdir(local): self._PushObject(os.path.join(local, f), os.path.join(remote, f)) - self.Shell('chmod 777 %s' % - cmd_helper.SingleQuote(os.path.join(remote, f))) + self.Shell( + 'chmod 777 %s' % cmd_helper.SingleQuote(os.path.join(remote, f))) else: parent_dir = remote[0:remote.rfind('/')] if parent_dir: @@ -76,17 +75,15 @@ class GceAdbWrapper(adb_wrapper.AdbWrapper): remote: Path on the instance filesystem. """ cmd = [ - 'scp', - '-r', - '-o', 'UserKnownHostsFile=/dev/null', - '-o', 'StrictHostKeyChecking=no', - local, + 'scp', '-r', '-o', 'UserKnownHostsFile=/dev/null', '-o', + 'StrictHostKeyChecking=no', local, 'root@%s:%s' % (self._instance_ip, remote) ] status, _ = cmd_helper.GetCmdStatusAndOutput(cmd) if status: raise device_errors.AdbCommandFailedError( - cmd, 'File not reachable on host: %s' % local, + cmd, + 'File not reachable on host: %s' % local, device_serial=str(self)) # override @@ -101,15 +98,18 @@ class GceAdbWrapper(adb_wrapper.AdbWrapper): 'scp', '-p', '-r', - '-o', 'UserKnownHostsFile=/dev/null', - '-o', 'StrictHostKeyChecking=no', + '-o', + 'UserKnownHostsFile=/dev/null', + '-o', + 'StrictHostKeyChecking=no', 'root@%s:%s' % (self._instance_ip, remote), local, ] status, _ = cmd_helper.GetCmdStatusAndOutput(cmd) if status: raise device_errors.AdbCommandFailedError( - cmd, 'File not reachable on host: %s' % local, + cmd, + 'File not reachable on host: %s' % local, device_serial=str(self)) try: @@ -117,12 +117,17 @@ class GceAdbWrapper(adb_wrapper.AdbWrapper): except (subprocess.CalledProcessError, IOError): logger.exception('Error when pulling files from android instance.') raise device_errors.AdbCommandFailedError( - cmd, 'File not reachable on host: %s' % local, + cmd, + 'File not reachable on host: %s' % local, device_serial=str(self)) # override - def Install(self, apk_path, forward_lock=False, reinstall=False, - sd_card=False, **kwargs): + def Install(self, + apk_path, + forward_lock=False, + reinstall=False, + sd_card=False, + **kwargs): """Installs an apk on the gce instance Args: diff --git a/catapult/devil/devil/android/sdk/intent.py b/catapult/devil/devil/android/sdk/intent.py index cdefb463..69c7d900 100644 --- a/catapult/devil/devil/android/sdk/intent.py +++ b/catapult/devil/devil/android/sdk/intent.py @@ -1,7 +1,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Manages intents and associated information. This is generally intended to be used with functions that calls Android's @@ -25,10 +24,15 @@ def _bitwise_or(flags): class Intent(object): - - def __init__(self, action='android.intent.action.VIEW', activity=None, - category=None, component=None, data=None, extras=None, - flags=None, package=None): + def __init__(self, + action='android.intent.action.VIEW', + activity=None, + category=None, + component=None, + data=None, + extras=None, + flags=None, + package=None): """Creates an Intent. Args: diff --git a/catapult/devil/devil/android/sdk/keyevent.py b/catapult/devil/devil/android/sdk/keyevent.py index 657dc963..61d61c84 100644 --- a/catapult/devil/devil/android/sdk/keyevent.py +++ b/catapult/devil/devil/android/sdk/keyevent.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Android KeyEvent constants. http://developer.android.com/reference/android/view/KeyEvent.html diff --git a/catapult/devil/devil/android/sdk/shared_prefs.py b/catapult/devil/devil/android/sdk/shared_prefs.py index c8c82b4e..7b12bf54 100644 --- a/catapult/devil/devil/android/sdk/shared_prefs.py +++ b/catapult/devil/devil/android/sdk/shared_prefs.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Helper object to read and modify Shared Preferences from Android apps. See e.g.: @@ -17,7 +16,6 @@ from devil.android.sdk import version_codes logger = logging.getLogger(__name__) - _XML_DECLARATION = "<?xml version='1.0' encoding='utf-8' standalone='yes' ?>\n" @@ -161,12 +159,14 @@ class StringSetPref(StringPref): ElementTree.SubElement(self._elem, 'string').text = str(item) -_PREF_TYPES = {c.tag_name: c for c in [BooleanPref, FloatPref, IntPref, - LongPref, StringPref, StringSetPref]} +_PREF_TYPES = { + c.tag_name: c + for c in + [BooleanPref, FloatPref, IntPref, LongPref, StringPref, StringSetPref] +} class SharedPrefs(object): - def __init__(self, device, package, filename, use_encrypted_path=False): """Helper object to read and update "Shared Prefs" of Android apps. @@ -224,8 +224,10 @@ class SharedPrefs(object): def __repr__(self): """Get a useful printable representation of the object.""" return '<{cls} file {filename} for {package} on {device}>'.format( - cls=type(self).__name__, filename=self.filename, package=self.package, - device=str(self._device)) + cls=type(self).__name__, + filename=self.filename, + package=self.package, + device=str(self._device)) def __str__(self): """Get the underlying xml document as a string.""" @@ -292,15 +294,16 @@ class SharedPrefs(object): return self._device.RunShellCommand( ['mkdir', '-p', posixpath.dirname(self.path)], - as_root=True, check_return=True) + as_root=True, + check_return=True) self._device.WriteFile(self.path, str(self), as_root=True) # Creating the directory/file can cause issues with SELinux if they did # not already exist. As a workaround, apply the package's security context # to the shared_prefs directory, which mimics the behavior of a file # created by the app itself if self._device.build_version_sdk >= version_codes.MARSHMALLOW: - security_context = self._device.GetSecurityContextForPackage(self.package, - encrypted=self._encrypted) + security_context = self._device.GetSecurityContextForPackage( + self.package, encrypted=self._encrypted) if security_context is None: raise device_errors.CommandFailedError( 'Failed to get security context for %s' % self.package) @@ -310,11 +313,11 @@ class SharedPrefs(object): # Ensure that there isn't both an encrypted and unencrypted version of the # file on the device at the same time. if self._device.build_version_sdk >= version_codes.NOUGAT: - remove_path = (self._unencrypted_path if self._encrypted - else self._encrypted_path) + remove_path = (self._unencrypted_path + if self._encrypted else self._encrypted_path) if self._device.PathExists(remove_path, as_root=True): logging.warning('Found an equivalent shared prefs file at %s, removing', - remove_path) + remove_path) self._device.RemovePath(remove_path, as_root=True) self._device.KillAll(self.package, exact=True, as_root=True, quiet=True) @@ -431,8 +434,8 @@ class SharedPrefs(object): pref = pref_cls(self._GetChild(key)) old_value = pref.get() except KeyError: - pref = pref_cls(ElementTree.SubElement( - self.xml, pref_cls.tag_name, {'name': key})) + pref = pref_cls( + ElementTree.SubElement(self.xml, pref_cls.tag_name, {'name': key})) old_value = None if old_value != value: pref.set(value) diff --git a/catapult/devil/devil/android/sdk/shared_prefs_test.py b/catapult/devil/devil/android/sdk/shared_prefs_test.py index 08bbb467..7374c892 100755 --- a/catapult/devil/devil/android/sdk/shared_prefs_test.py +++ b/catapult/devil/devil/android/sdk/shared_prefs_test.py @@ -2,7 +2,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Unit tests for the contents of shared_prefs.py (mostly SharedPrefs). """ @@ -18,7 +17,6 @@ from devil.android.sdk import version_codes with devil_env.SysPath(devil_env.PYMOCK_PATH): import mock # pylint: disable=import-error - INITIAL_XML = ("<?xml version='1.0' encoding='utf-8' standalone='yes' ?>\n" '<map>\n' ' <int name="databaseVersion" value="107" />\n' @@ -48,17 +46,19 @@ def MockDeviceWithFiles(files=None): class SharedPrefsTest(unittest.TestCase): - def setUp(self): self.device = MockDeviceWithFiles({ - '/data/data/com.some.package/shared_prefs/prefs.xml': INITIAL_XML}) - self.expected_data = {'databaseVersion': 107, - 'featureEnabled': False, - 'someHashValue': '249b3e5af13d4db2'} + '/data/data/com.some.package/shared_prefs/prefs.xml': INITIAL_XML + }) + self.expected_data = { + 'databaseVersion': 107, + 'featureEnabled': False, + 'someHashValue': '249b3e5af13d4db2' + } def testPropertyLifetime(self): - prefs = shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') + prefs = shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') self.assertEquals(len(prefs), 0) # collection is empty before loading prefs.SetInt('myValue', 444) self.assertEquals(len(prefs), 1) @@ -71,8 +71,8 @@ class SharedPrefsTest(unittest.TestCase): prefs.GetInt('myValue') def testPropertyType(self): - prefs = shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') + prefs = shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') prefs.SetInt('myValue', 444) self.assertEquals(prefs.PropertyType('myValue'), 'int') with self.assertRaises(TypeError): @@ -81,8 +81,8 @@ class SharedPrefsTest(unittest.TestCase): prefs.SetString('myValue', 'hello') def testLoad(self): - prefs = shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') + prefs = shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') self.assertEquals(len(prefs), 0) # collection is empty before loading prefs.Load() self.assertEquals(len(prefs), len(self.expected_data)) @@ -90,8 +90,8 @@ class SharedPrefsTest(unittest.TestCase): self.assertFalse(prefs.changed) def testClear(self): - prefs = shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') + prefs = shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') prefs.Load() self.assertEquals(prefs.AsDict(), self.expected_data) self.assertFalse(prefs.changed) @@ -102,8 +102,8 @@ class SharedPrefsTest(unittest.TestCase): def testCommit(self): type(self.device).build_version_sdk = mock.PropertyMock( return_value=version_codes.LOLLIPOP_MR1) - prefs = shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'other_prefs.xml') + prefs = shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'other_prefs.xml') self.assertFalse(self.device.FileExists(prefs.path)) # file does not exist prefs.Load() self.assertEquals(len(prefs), 0) # file did not exist, collection is empty @@ -115,54 +115,58 @@ class SharedPrefsTest(unittest.TestCase): self.assertTrue(prefs.changed) prefs.Commit() self.assertTrue(self.device.FileExists(prefs.path)) # should exist now - self.device.KillAll.assert_called_once_with(prefs.package, exact=True, - as_root=True, quiet=True) + self.device.KillAll.assert_called_once_with( + prefs.package, exact=True, as_root=True, quiet=True) self.assertFalse(prefs.changed) - prefs = shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'other_prefs.xml') + prefs = shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'other_prefs.xml') self.assertEquals(len(prefs), 0) # collection is empty before loading prefs.Load() - self.assertEquals(prefs.AsDict(), { - 'magicNumber': 42, - 'myMetric': 3.14, - 'bigNumner': 6000000000, - 'apps': ['gmail', 'chrome', 'music']}) # data survived roundtrip + self.assertEquals( + prefs.AsDict(), { + 'magicNumber': 42, + 'myMetric': 3.14, + 'bigNumner': 6000000000, + 'apps': ['gmail', 'chrome', 'music'] + }) # data survived roundtrip def testForceCommit(self): - prefs = shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') + prefs = shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') prefs.Load() new_xml = 'Not valid XML' self.device.WriteFile('/data/data/com.some.package/shared_prefs/prefs.xml', - new_xml) + new_xml) prefs.Commit() # Since we didn't change anything, Commit() should be a no-op. - self.assertEquals(self.device.ReadFile( - '/data/data/com.some.package/shared_prefs/prefs.xml'), new_xml) + self.assertEquals( + self.device.ReadFile( + '/data/data/com.some.package/shared_prefs/prefs.xml'), new_xml) prefs.Commit(force_commit=True) # Forcing the commit should restore the originally read XML. - self.assertEquals(self.device.ReadFile( - '/data/data/com.some.package/shared_prefs/prefs.xml'), INITIAL_XML) + self.assertEquals( + self.device.ReadFile( + '/data/data/com.some.package/shared_prefs/prefs.xml'), INITIAL_XML) def testAsContextManager_onlyReads(self): - with shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') as prefs: + with shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') as prefs: self.assertEquals(prefs.AsDict(), self.expected_data) # loaded and ready self.assertEquals(self.device.WriteFile.call_args_list, []) # did not write def testAsContextManager_readAndWrite(self): type(self.device).build_version_sdk = mock.PropertyMock( return_value=version_codes.LOLLIPOP_MR1) - with shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') as prefs: + with shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') as prefs: prefs.SetBoolean('featureEnabled', True) prefs.Remove('someHashValue') prefs.SetString('newString', 'hello') self.assertTrue(self.device.WriteFile.called) # did write - with shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') as prefs: + with shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') as prefs: # changes persisted self.assertTrue(prefs.GetBoolean('featureEnabled')) self.assertFalse(prefs.HasProperty('someHashValue')) @@ -171,32 +175,35 @@ class SharedPrefsTest(unittest.TestCase): def testAsContextManager_commitAborted(self): with self.assertRaises(TypeError): - with shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') as prefs: + with shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') as prefs: prefs.SetBoolean('featureEnabled', True) prefs.Remove('someHashValue') prefs.SetString('newString', 'hello') prefs.SetInt('newString', 123) # oops! self.assertEquals(self.device.WriteFile.call_args_list, []) # did not write - with shared_prefs.SharedPrefs( - self.device, 'com.some.package', 'prefs.xml') as prefs: + with shared_prefs.SharedPrefs(self.device, 'com.some.package', + 'prefs.xml') as prefs: # contents were not modified self.assertEquals(prefs.AsDict(), self.expected_data) def testEncryptedPath(self): type(self.device).build_version_sdk = mock.PropertyMock( return_value=version_codes.MARSHMALLOW) - with shared_prefs.SharedPrefs(self.device, 'com.some.package', - 'prefs.xml', use_encrypted_path=True) as prefs: + with shared_prefs.SharedPrefs( + self.device, 'com.some.package', 'prefs.xml', + use_encrypted_path=True) as prefs: self.assertTrue(prefs.path.startswith('/data/data')) type(self.device).build_version_sdk = mock.PropertyMock( return_value=version_codes.NOUGAT) - with shared_prefs.SharedPrefs(self.device, 'com.some.package', - 'prefs.xml', use_encrypted_path=True) as prefs: + with shared_prefs.SharedPrefs( + self.device, 'com.some.package', 'prefs.xml', + use_encrypted_path=True) as prefs: self.assertTrue(prefs.path.startswith('/data/user_de/0')) + if __name__ == '__main__': logging.getLogger().setLevel(logging.DEBUG) unittest.main(verbosity=2) diff --git a/catapult/devil/devil/android/sdk/split_select.py b/catapult/devil/devil/android/sdk/split_select.py index 6c3d231a..6b46e576 100644 --- a/catapult/devil/devil/android/sdk/split_select.py +++ b/catapult/devil/devil/android/sdk/split_select.py @@ -1,16 +1,14 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """This module wraps Android's split-select tool.""" from devil.android.sdk import build_tools from devil.utils import cmd_helper from devil.utils import lazy - -_split_select_path = lazy.WeakConstant( - lambda: build_tools.GetPath('split-select')) +_split_select_path = lazy.WeakConstant(lambda: build_tools.GetPath( + 'split-select')) def _RunSplitSelectCmd(args): @@ -37,11 +35,9 @@ def _SplitConfig(device, allow_cached_props=False): device: A DeviceUtils object. allow_cached_props: Whether to use cached values for device properties. """ - return ('%s-r%s-%s:%s' % - (device.GetLanguage(cache=allow_cached_props), - device.GetCountry(cache=allow_cached_props), - device.screen_density, - device.product_cpu_abi)) + return ('%s-r%s-%s:%s' % (device.GetLanguage(cache=allow_cached_props), + device.GetCountry(cache=allow_cached_props), + device.screen_density, device.product_cpu_abi)) def SelectSplits(device, base_apk, split_apks, allow_cached_props=False): diff --git a/catapult/devil/devil/android/sdk/version_codes.py b/catapult/devil/devil/android/sdk/version_codes.py index 29c7285e..943b9d3f 100644 --- a/catapult/devil/devil/android/sdk/version_codes.py +++ b/catapult/devil/devil/android/sdk/version_codes.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Android SDK version codes. http://developer.android.com/reference/android/os/Build.VERSION_CODES.html @@ -20,3 +19,4 @@ NOUGAT_MR1 = 25 OREO = 26 OREO_MR1 = 27 PIE = 28 +Q = 29 diff --git a/catapult/devil/devil/android/settings.py b/catapult/devil/devil/android/settings.py index 1713be46..3cb1002d 100644 --- a/catapult/devil/devil/android/settings.py +++ b/catapult/devil/devil/android/settings.py @@ -12,111 +12,110 @@ _ALTERNATE_LOCK_SCREEN_SETTINGS_PATH = ( PASSWORD_QUALITY_UNSPECIFIED = '0' _COMPATIBLE_BUILD_TYPES = ['userdebug', 'eng'] - ENABLE_LOCATION_SETTINGS = [ - # Note that setting these in this order is required in order for all of - # them to take and stick through a reboot. - ('com.google.settings/partner', [ - ('use_location_for_services', 1), - ]), - ('settings/secure', [ - # Ensure Geolocation is enabled and allowed for tests. - ('location_providers_allowed', 'gps,network'), - ]), - ('com.google.settings/partner', [ - ('network_location_opt_in', 1), - ]) + # Note that setting these in this order is required in order for all of + # them to take and stick through a reboot. + ('com.google.settings/partner', [ + ('use_location_for_services', 1), + ]), + ( + 'settings/secure', + [ + # Ensure Geolocation is enabled and allowed for tests. + ('location_providers_allowed', 'gps,network'), + ]), + ('com.google.settings/partner', [ + ('network_location_opt_in', 1), + ]) ] DISABLE_LOCATION_SETTINGS = [ - ('com.google.settings/partner', [ - ('use_location_for_services', 0), - ]), - ('settings/secure', [ - # Ensure Geolocation is disabled. - ('location_providers_allowed', ''), - ]), + ('com.google.settings/partner', [ + ('use_location_for_services', 0), + ]), + ( + 'settings/secure', + [ + # Ensure Geolocation is disabled. + ('location_providers_allowed', ''), + ]), ] ENABLE_MOCK_LOCATION_SETTINGS = [ - ('settings/secure', [ - ('mock_location', 1), - ]), + ('settings/secure', [ + ('mock_location', 1), + ]), ] DISABLE_MOCK_LOCATION_SETTINGS = [ - ('settings/secure', [ - ('mock_location', 0), - ]), + ('settings/secure', [ + ('mock_location', 0), + ]), ] DETERMINISTIC_DEVICE_SETTINGS = [ - ('settings/global', [ - ('assisted_gps_enabled', 0), - - # Disable "auto time" and "auto time zone" to avoid network-provided time - # to overwrite the device's datetime and timezone synchronized from host - # when running tests later. See b/6569849. - ('auto_time', 0), - ('auto_time_zone', 0), - - ('development_settings_enabled', 1), - - # Flag for allowing ActivityManagerService to send ACTION_APP_ERROR intents - # on application crashes and ANRs. If this is disabled, the crash/ANR dialog - # will never display the "Report" button. - # Type: int ( 0 = disallow, 1 = allow ) - ('send_action_app_error', 0), - - ('stay_on_while_plugged_in', 3), - - ('verifier_verify_adb_installs', 0), - - ('window_animation_scale', 0), - ]), - ('settings/secure', [ - ('allowed_geolocation_origins', - 'http://www.google.co.uk http://www.google.com'), - - # Ensure that we never get random dialogs like "Unfortunately the process - # android.process.acore has stopped", which steal the focus, and make our - # automation fail (because the dialog steals the focus then mistakenly - # receives the injected user input events). - ('anr_show_background', 0), - - ('lockscreen.disabled', 1), - - ('screensaver_enabled', 0), - - ('skip_first_use_hints', 1), - ]), - ('settings/system', [ - # Don't want devices to accidentally rotate the screen as that could - # affect performance measurements. - ('accelerometer_rotation', 0), - - ('lockscreen.disabled', 1), - - # Turn down brightness and disable auto-adjust so that devices run cooler. - ('screen_brightness', 5), - ('screen_brightness_mode', 0), - - ('user_rotation', 0), - - ('window_animation_scale', 0), - ]), + ( + 'settings/global', + [ + ('assisted_gps_enabled', 0), + + # Disable "auto time" and "auto time zone" to avoid network-provided + # time to overwrite the device's datetime and timezone synchronized + # from host when running tests later. See b/6569849. + ('auto_time', 0), + ('auto_time_zone', 0), + ('development_settings_enabled', 1), + + # Flag for allowing ActivityManagerService to send ACTION_APP_ERROR + # intens on application crashes and ANRs. If this is disabled, the + # crash/ANR dialog will never display the "Report" button. + # Type: int ( 0 = disallow, 1 = allow ) + ('send_action_app_error', 0), + ('stay_on_while_plugged_in', 3), + ('verifier_verify_adb_installs', 0), + ('window_animation_scale', 0), + ]), + ( + 'settings/secure', + [ + ('allowed_geolocation_origins', + 'http://www.google.co.uk http://www.google.com'), + + # Ensure that we never get random dialogs like "Unfortunately the + # process android.process.acore has stopped", which steal the focus, + # and make our automation fail (because the dialog steals the focus + # then mistakenly receives the injected user input events). + ('anr_show_background', 0), + ('lockscreen.disabled', 1), + ('screensaver_enabled', 0), + ('skip_first_use_hints', 1), + ]), + ( + 'settings/system', + [ + # Don't want devices to accidentally rotate the screen as that could + # affect performance measurements. + ('accelerometer_rotation', 0), + ('lockscreen.disabled', 1), + + # Turn down brightness and disable auto-adjust so that devices run + # cooler. + ('screen_brightness', 5), + ('screen_brightness_mode', 0), + ('user_rotation', 0), + ('window_animation_scale', 0), + ]), ] NETWORK_DISABLED_SETTINGS = [ - ('settings/global', [ - ('airplane_mode_on', 1), - ('wifi_on', 0), - ]), + ('settings/global', [ + ('airplane_mode_on', 1), + ('wifi_on', 0), + ]), ] class ContentSettings(dict): - """A dict interface to interact with device content settings. System properties are key/value pairs as exposed by adb shell content. @@ -143,18 +142,24 @@ class ContentSettings(dict): def iteritems(self): for row in self._device.RunShellCommand( - ['content', 'query', '--uri', 'content://%s' % self._table], - check_return=True, as_root=True): + ['content', 'query', '--uri', + 'content://%s' % self._table], + check_return=True, + as_root=True): key, value = _ParseContentRow(row) if not key: continue yield key, value def __getitem__(self, key): - query_row = self._device.RunShellCommand( - ['content', 'query', '--uri', 'content://%s' % self._table, - '--where', "name='%s'" % key], - check_return=True, as_root=True, single_line=True) + query_row = self._device.RunShellCommand([ + 'content', 'query', '--uri', + 'content://%s' % self._table, '--where', + "name='%s'" % key + ], + check_return=True, + as_root=True, + single_line=True) parsed_key, parsed_value = _ParseContentRow(query_row) if parsed_key is None: raise KeyError('key=%s not found' % key) @@ -164,23 +169,32 @@ class ContentSettings(dict): def __setitem__(self, key, value): if key in self: - self._device.RunShellCommand( - ['content', 'update', '--uri', 'content://%s' % self._table, - '--bind', 'value:%s:%s' % (self._GetTypeBinding(value), value), - '--where', "name='%s'" % key], - check_return=True, as_root=True) + self._device.RunShellCommand([ + 'content', 'update', '--uri', + 'content://%s' % self._table, '--bind', + 'value:%s:%s' % (self._GetTypeBinding(value), value), '--where', + "name='%s'" % key + ], + check_return=True, + as_root=True) else: - self._device.RunShellCommand( - ['content', 'insert', '--uri', 'content://%s' % self._table, - '--bind', 'name:%s:%s' % (self._GetTypeBinding(key), key), - '--bind', 'value:%s:%s' % (self._GetTypeBinding(value), value)], - check_return=True, as_root=True) + self._device.RunShellCommand([ + 'content', 'insert', '--uri', + 'content://%s' % self._table, '--bind', + 'name:%s:%s' % (self._GetTypeBinding(key), key), '--bind', + 'value:%s:%s' % (self._GetTypeBinding(value), value) + ], + check_return=True, + as_root=True) def __delitem__(self, key): - self._device.RunShellCommand( - ['content', 'delete', '--uri', 'content://%s' % self._table, - '--bind', 'name:%s:%s' % (self._GetTypeBinding(key), key)], - check_return=True, as_root=True) + self._device.RunShellCommand([ + 'content', 'delete', '--uri', + 'content://%s' % self._table, '--bind', + 'name:%s:%s' % (self._GetTypeBinding(key), key) + ], + check_return=True, + as_root=True) def ConfigureContentSettings(device, desired_settings): @@ -259,14 +273,15 @@ def SetLockScreenSettings(device): delete from '%(table)s' where %(primary_key)s='%(primary_value)s'; insert into '%(table)s' (%(columns)s) values (%(values)s); commit transaction;""" % { - 'table': table, - 'primary_key': columns[0], - 'primary_value': values[0], - 'columns': ', '.join(columns), - 'values': ', '.join(["'%s'" % value for value in values]) + 'table': table, + 'primary_key': columns[0], + 'primary_value': values[0], + 'columns': ', '.join(columns), + 'values': ', '.join(["'%s'" % value for value in values]) } - output_msg = device.RunShellCommand( - ['sqlite3', db, cmd], check_return=True, as_root=True) + output_msg = device.RunShellCommand(['sqlite3', db, cmd], + check_return=True, + as_root=True) if output_msg: logger.info(' '.join(output_msg)) diff --git a/catapult/devil/devil/android/tools/adb_run_shell_cmd.py b/catapult/devil/devil/android/tools/adb_run_shell_cmd.py index 6edd5606..e6b05aa2 100755 --- a/catapult/devil/devil/android/tools/adb_run_shell_cmd.py +++ b/catapult/devil/devil/android/tools/adb_run_shell_cmd.py @@ -10,8 +10,8 @@ import sys if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil.android import device_utils from devil.android.tools import script_common @@ -26,17 +26,16 @@ def main(): script_common.AddDeviceArguments(parser) script_common.AddEnvironmentArguments(parser) parser.add_argument('--as-root', action='store_true', help='Run as root.') - parser.add_argument('--json-output', - help='File to dump json output to.') + parser.add_argument('--json-output', help='File to dump json output to.') args = parser.parse_args() logging_common.InitializeLogging(args) script_common.InitializeEnvironment(args) - devices = script_common.GetDevices(args.devices, args.blacklist_file) + devices = script_common.GetDevices(args.devices, args.denylist_file) p_out = (device_utils.DeviceUtils.parallel(devices).RunShellCommand( - args.cmd, large_output=True, as_root=args.as_root, check_return=True) - .pGet(None)) + args.cmd, large_output=True, as_root=args.as_root, + check_return=True).pGet(None)) data = {} for device, output in zip(devices, p_out): diff --git a/catapult/devil/devil/android/tools/cpufreq.py b/catapult/devil/devil/android/tools/cpufreq.py index 6ce0affd..f33542bf 100755 --- a/catapult/devil/devil/android/tools/cpufreq.py +++ b/catapult/devil/devil/android/tools/cpufreq.py @@ -2,7 +2,6 @@ # Copyright 2016 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A script to manipulate device CPU frequency.""" import argparse @@ -12,8 +11,8 @@ import sys if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil.android import device_utils from devil.android.perf import perf_control @@ -43,15 +42,16 @@ def main(raw_args): logging_common.AddLoggingArguments(parser) script_common.AddEnvironmentArguments(parser) parser.add_argument( - '--device', dest='devices', action='append', default=[], + '--device', + dest='devices', + action='append', + default=[], help='Devices for which the governor should be set. Defaults to all.') subparsers = parser.add_subparsers() set_governor = subparsers.add_parser('set-governor') - set_governor.add_argument( - 'governor', - help='Desired CPU governor.') + set_governor.add_argument('governor', help='Desired CPU governor.') set_governor.set_defaults(func=SetScalingGovernor) get_governor = subparsers.add_parser('get-governor') diff --git a/catapult/devil/devil/android/tools/device_monitor.py b/catapult/devil/devil/android/tools/device_monitor.py index 565f8658..6128dae8 100755 --- a/catapult/devil/devil/android/tools/device_monitor.py +++ b/catapult/devil/devil/android/tools/device_monitor.py @@ -2,7 +2,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Launches a daemon to monitor android device temperatures & status. This script will repeatedly poll the given devices for their temperatures and @@ -22,24 +21,23 @@ import time if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil.android import battery_utils -from devil.android import device_blacklist +from devil.android import device_denylist from devil.android import device_errors from devil.android import device_utils from devil.android.tools import script_common - # Various names of sensors used to measure cpu temp CPU_TEMP_SENSORS = [ - # most nexus devices - 'tsens_tz_sensor0', - # android one - 'mtktscpu', - # nexus 9 - 'CPU-therm', + # most nexus devices + 'tsens_tz_sensor0', + # android one + 'mtktscpu', + # nexus 9 + 'CPU-therm', ] DEVICE_FILE_VERSION = 1 @@ -47,7 +45,8 @@ DEVICE_FILE = os.path.join( os.path.expanduser('~'), '.android', '%s__android_device_status.json' % socket.gethostname().split('.')[0]) -MEM_INFO_REGEX = re.compile(r'.*?\:\s*(\d+)\s*kB') # ex: 'MemTotal: 185735 kB' +MEM_INFO_REGEX = re.compile( + r'.*?\:\s*(\d+)\s*kB') # ex: 'MemTotal: 185735 kB' def get_device_status_unsafe(device): @@ -129,14 +128,16 @@ def get_device_status_unsafe(device): files = [] try: files = device.RunShellCommand( - 'grep -lE "%s" /sys/class/thermal/thermal_zone*/type' % '|'.join( - CPU_TEMP_SENSORS), shell=True, check_return=True) + 'grep -lE "%s" /sys/class/thermal/thermal_zone*/type' % + '|'.join(CPU_TEMP_SENSORS), + shell=True, + check_return=True) except device_errors.AdbShellCommandFailedError: logging.exception('Unable to list thermal sensors.') for f in files: try: sensor_name = device.ReadFile(f).strip() - temp = float(device.ReadFile(f[:-4] + 'temp').strip()) # s/type^/temp + temp = float(device.ReadFile(f[:-4] + 'temp').strip()) # s/type^/temp status['temp'][sensor_name] = temp except (device_errors.AdbShellCommandFailedError, ValueError): logging.exception('Unable to read thermal sensor %s', f) @@ -144,7 +145,7 @@ def get_device_status_unsafe(device): # Uptime try: uptimes = device.ReadFile('/proc/uptime').split() - status['uptime'] = float(uptimes[0]) # Take the first field (actual uptime) + status['uptime'] = float(uptimes[0]) # Take the first field (actual uptime) except (device_errors.AdbShellCommandFailedError, ValueError): logging.exception('Unable to read /proc/uptime') @@ -167,24 +168,26 @@ def get_device_status(device): return status -def get_all_status(blacklist): +def get_all_status(denylist): status_dict = { 'version': DEVICE_FILE_VERSION, 'devices': {}, } - healthy_devices = device_utils.DeviceUtils.HealthyDevices(blacklist) + healthy_devices = device_utils.DeviceUtils.HealthyDevices(denylist) parallel_devices = device_utils.DeviceUtils.parallel(healthy_devices) results = parallel_devices.pMap(get_device_status).pGet(None) status_dict['devices'] = { - device.serial: result for device, result in zip(healthy_devices, results) + device.serial: result + for device, result in zip(healthy_devices, results) } - if blacklist: - for device, reason in blacklist.Read().iteritems(): + if denylist: + for device, reason in denylist.Read().iteritems(): status_dict['devices'][device] = { - 'state': reason.get('reason', 'blacklisted')} + 'state': reason.get('reason', 'denylisted') + } status_dict['timestamp'] = time.time() return status_dict @@ -194,33 +197,37 @@ def main(argv): """Launches the device monitor. Polls the devices for their battery and cpu temperatures and scans the - blacklist file every 60 seconds and dumps the data to DEVICE_FILE. + denylist file every 60 seconds and dumps the data to DEVICE_FILE. """ - parser = argparse.ArgumentParser( - description='Launches the device monitor.') + parser = argparse.ArgumentParser(description='Launches the device monitor.') script_common.AddEnvironmentArguments(parser) - parser.add_argument('--blacklist-file', help='Path to device blacklist file.') + parser.add_argument('--denylist-file', help='Path to device denylist file.') + # TODO(crbug.com/1097306): Remove this once chromium_android/api.py stops + # using it. + parser.add_argument('--blacklist-file', + dest='denylist_file', + help=argparse.SUPPRESS) args = parser.parse_args(argv) logger = logging.getLogger() logger.setLevel(logging.DEBUG) handler = logging.handlers.RotatingFileHandler( '/tmp/device_monitor.log', maxBytes=10 * 1024 * 1024, backupCount=5) - fmt = logging.Formatter('%(asctime)s %(levelname)s %(message)s', - datefmt='%y%m%d %H:%M:%S') + fmt = logging.Formatter( + '%(asctime)s %(levelname)s %(message)s', datefmt='%y%m%d %H:%M:%S') handler.setFormatter(fmt) logger.addHandler(handler) script_common.InitializeEnvironment(args) - blacklist = (device_blacklist.Blacklist(args.blacklist_file) - if args.blacklist_file else None) + denylist = (device_denylist.Denylist(args.denylist_file) + if args.denylist_file else None) - logging.info('Device monitor running with pid %d, adb: %s, blacklist: %s', - os.getpid(), args.adb_path, args.blacklist_file) + logging.info('Device monitor running with pid %d, adb: %s, denylist: %s', + os.getpid(), args.adb_path, args.denylist_file) while True: start = time.time() - status_dict = get_all_status(blacklist) + status_dict = get_all_status(denylist) with open(DEVICE_FILE, 'wb') as f: json.dump(status_dict, f, indent=2, sort_keys=True) logging.info('Got status of all devices in %.2fs.', time.time() - start) diff --git a/catapult/devil/devil/android/tools/device_monitor_test.py b/catapult/devil/devil/android/tools/device_monitor_test.py index 2cb0dd28..8082d26e 100755 --- a/catapult/devil/devil/android/tools/device_monitor_test.py +++ b/catapult/devil/devil/android/tools/device_monitor_test.py @@ -9,8 +9,8 @@ import unittest if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil import devil_env from devil.android import device_errors @@ -22,20 +22,26 @@ with devil_env.SysPath(devil_env.PYMOCK_PATH): class DeviceMonitorTest(unittest.TestCase): - def setUp(self): - self.device = mock.Mock(spec=device_utils.DeviceUtils, - serial='device_cereal', build_id='abc123', build_product='clownfish', + self.device = mock.Mock( + spec=device_utils.DeviceUtils, + serial='device_cereal', + build_id='abc123', + build_product='clownfish', GetIMEI=lambda: '123456789') self.file_contents = { - '/proc/meminfo': """ + '/proc/meminfo': + """ MemTotal: 1234567 kB MemFree: 1000000 kB MemUsed: 234567 kB """, - '/sys/class/thermal/thermal_zone0/type': 'CPU-therm', - '/sys/class/thermal/thermal_zone0/temp': '30', - '/proc/uptime': '12345 99999', + '/sys/class/thermal/thermal_zone0/type': + 'CPU-therm', + '/sys/class/thermal/thermal_zone0/temp': + '30', + '/proc/uptime': + '12345 99999', } self.device.ReadFile = mock.MagicMock( side_effect=lambda file_name: self.file_contents[file_name]) @@ -56,31 +62,33 @@ class DeviceMonitorTest(unittest.TestCase): self.device.RunShellCommand = mock.MagicMock(side_effect=mock_run_shell) self.battery = mock.Mock() - self.battery.GetBatteryInfo = mock.MagicMock( - return_value={'level': '80', 'temperature': '123'}) + self.battery.GetBatteryInfo = mock.MagicMock(return_value={ + 'level': '80', + 'temperature': '123' + }) self.expected_status = { - 'device_cereal': { - 'processes': 5, - 'temp': { - 'CPU-therm': 30.0 - }, - 'battery': { - 'temperature': 123, - 'level': 80 - }, - 'uptime': 12345.0, - 'mem': { - 'total': 1234567, - 'free': 1000000 - }, - 'build': { - 'build.id': 'abc123', - 'product.device': 'clownfish', - }, - 'imei': '123456789', - 'state': 'available', - } + 'device_cereal': { + 'processes': 5, + 'temp': { + 'CPU-therm': 30.0 + }, + 'battery': { + 'temperature': 123, + 'level': 80 + }, + 'uptime': 12345.0, + 'mem': { + 'total': 1234567, + 'free': 1000000 + }, + 'build': { + 'build.id': 'abc123', + 'product.device': 'clownfish', + }, + 'imei': '123456789', + 'state': 'available', + } } @mock.patch('devil.android.battery_utils.BatteryUtils') @@ -98,8 +106,10 @@ class DeviceMonitorTest(unittest.TestCase): get_devices.return_value = [self.device] get_battery.return_value = self.battery broken_battery_info = mock.Mock() - broken_battery_info.GetBatteryInfo = mock.MagicMock( - return_value={'level': '-1', 'temperature': 'not_a_number'}) + broken_battery_info.GetBatteryInfo = mock.MagicMock(return_value={ + 'level': '-1', + 'temperature': 'not_a_number' + }) get_battery.return_value = broken_battery_info # Should be same status dict but without battery stats. @@ -141,18 +151,20 @@ class DeviceMonitorTest(unittest.TestCase): @mock.patch('devil.android.battery_utils.BatteryUtils') @mock.patch('devil.android.device_utils.DeviceUtils.HealthyDevices') - def test_getStatsWithBlacklist(self, get_devices, get_battery): + def test_getStatsWithDenylist(self, get_devices, get_battery): get_devices.return_value = [self.device] get_battery.return_value = self.battery - blacklist = mock.Mock() - blacklist.Read = mock.MagicMock( - return_value={'bad_device': {'reason': 'offline'}}) + denylist = mock.Mock() + denylist.Read = mock.MagicMock( + return_value={'bad_device': { + 'reason': 'offline' + }}) - # Should be same status dict but with extra blacklisted device. + # Should be same status dict but with extra denylisted device. expected_status = self.expected_status.copy() expected_status['bad_device'] = {'state': 'offline'} - status = device_monitor.get_all_status(blacklist) + status = device_monitor.get_all_status(denylist) self.assertEquals(expected_status, status['devices']) @mock.patch('devil.android.battery_utils.BatteryUtils') diff --git a/catapult/devil/devil/android/tools/device_recovery.py b/catapult/devil/devil/android/tools/device_recovery.py index 8050e6fe..f25c5a25 100755 --- a/catapult/devil/devil/android/tools/device_recovery.py +++ b/catapult/devil/devil/android/tools/device_recovery.py @@ -2,7 +2,6 @@ # Copyright 2016 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A script to recover devices in a known bad state.""" import argparse @@ -16,9 +15,9 @@ import psutil if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) -from devil.android import device_blacklist + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) +from devil.android import device_denylist from devil.android import device_errors from devil.android import device_utils from devil.android.sdk import adb_wrapper @@ -33,7 +32,6 @@ logger = logging.getLogger(__name__) from py_utils import modules_util - # Script depends on features from psutil version 2.0 or higher. modules_util.RequireVersion(psutil, '2.0') @@ -76,16 +74,16 @@ def TryAuth(device): """ possible_keys = glob.glob(os.path.join(adb_wrapper.ADB_HOST_KEYS_DIR, '*key')) if len(possible_keys) <= 1: - logger.warning( - 'Only %d ADB keys available. Not forcing auth.', len(possible_keys)) + logger.warning('Only %d ADB keys available. Not forcing auth.', + len(possible_keys)) return False KillAllAdb() adb_wrapper.AdbWrapper.StartServer(keys=possible_keys) new_state = device.adb.GetState() if new_state != 'device': - logger.error( - 'Auth failed. Device %s still stuck in %s.', str(device), new_state) + logger.error('Auth failed. Device %s still stuck in %s.', str(device), + new_state) return False # It worked! Now register the host's default ADB key on the device so we don't @@ -99,18 +97,16 @@ def TryAuth(device): pub_key_contents = f.read() try: device.WriteFile(adb_wrapper.ADB_KEYS_FILE, pub_key_contents, as_root=True) - except (device_errors.CommandTimeoutError, - device_errors.CommandFailedError, + except (device_errors.CommandTimeoutError, device_errors.CommandFailedError, device_errors.DeviceUnreachableError): logger.exception('Unable to write default ADB key to %s.', str(device)) return False return True -def RecoverDevice(device, blacklist, should_reboot=lambda device: True): - if device_status.IsBlacklisted(device.adb.GetDeviceSerial(), - blacklist): - logger.debug('%s is blacklisted, skipping recovery.', str(device)) +def RecoverDevice(device, denylist, should_reboot=lambda device: True): + if device_status.IsDenylisted(device.adb.GetDeviceSerial(), denylist): + logger.debug('%s is denylisted, skipping recovery.', str(device)) return if device.adb.GetState() == 'unauthorized' and TryAuth(device): @@ -120,17 +116,18 @@ def RecoverDevice(device, blacklist, should_reboot=lambda device: True): if should_reboot(device): try: device.WaitUntilFullyBooted(retries=0) - except (device_errors.CommandTimeoutError, - device_errors.CommandFailedError, + except (device_errors.CommandTimeoutError, device_errors.CommandFailedError, device_errors.DeviceUnreachableError): - logger.exception('Failure while waiting for %s. ' - 'Attempting to recover.', str(device)) + logger.exception( + 'Failure while waiting for %s. ' + 'Attempting to recover.', str(device)) try: try: device.Reboot(block=False, timeout=5, retries=0) except device_errors.CommandTimeoutError: - logger.warning('Timed out while attempting to reboot %s normally.' - 'Attempting alternative reboot.', str(device)) + logger.warning( + 'Timed out while attempting to reboot %s normally.' + 'Attempting alternative reboot.', str(device)) # The device drops offline before we can grab the exit code, so # we don't check for status. try: @@ -139,19 +136,20 @@ def RecoverDevice(device, blacklist, should_reboot=lambda device: True): # We are already in a failure mode, attempt to reboot regardless of # what device.adb.Root() returns. If the sysrq reboot fails an # exception willbe thrown at that level. - device.adb.Shell('echo b > /proc/sysrq-trigger', expect_status=None, - timeout=5, retries=0) + device.adb.Shell( + 'echo b > /proc/sysrq-trigger', + expect_status=None, + timeout=5, + retries=0) except (device_errors.CommandFailedError, device_errors.DeviceUnreachableError): logger.exception('Failed to reboot %s.', str(device)) - if blacklist: - blacklist.Extend([device.adb.GetDeviceSerial()], - reason='reboot_failure') + if denylist: + denylist.Extend([device.adb.GetDeviceSerial()], reason='reboot_failure') except device_errors.CommandTimeoutError: logger.exception('Timed out while rebooting %s.', str(device)) - if blacklist: - blacklist.Extend([device.adb.GetDeviceSerial()], - reason='reboot_timeout') + if denylist: + denylist.Extend([device.adb.GetDeviceSerial()], reason='reboot_timeout') try: device.WaitUntilFullyBooted( @@ -159,37 +157,34 @@ def RecoverDevice(device, blacklist, should_reboot=lambda device: True): except (device_errors.CommandFailedError, device_errors.DeviceUnreachableError): logger.exception('Failure while waiting for %s.', str(device)) - if blacklist: - blacklist.Extend([device.adb.GetDeviceSerial()], - reason='reboot_failure') + if denylist: + denylist.Extend([device.adb.GetDeviceSerial()], reason='reboot_failure') except device_errors.CommandTimeoutError: logger.exception('Timed out while waiting for %s.', str(device)) - if blacklist: - blacklist.Extend([device.adb.GetDeviceSerial()], - reason='reboot_timeout') + if denylist: + denylist.Extend([device.adb.GetDeviceSerial()], reason='reboot_timeout') -def RecoverDevices(devices, blacklist, enable_usb_reset=False): +def RecoverDevices(devices, denylist, enable_usb_reset=False): """Attempts to recover any inoperable devices in the provided list. Args: devices: The list of devices to attempt to recover. - blacklist: The current device blacklist, which will be used then + denylist: The current device denylist, which will be used then reset. """ - statuses = device_status.DeviceStatus(devices, blacklist) + statuses = device_status.DeviceStatus(devices, denylist) should_restart_usb = set( status['serial'] for status in statuses - if (not status['usb_status'] - or status['adb_status'] in ('offline', 'missing'))) - should_restart_adb = should_restart_usb.union(set( - status['serial'] for status in statuses - if status['adb_status'] == 'unauthorized')) - should_reboot_device = should_restart_usb.union(set( - status['serial'] for status in statuses - if status['blacklisted'])) + if (not status['usb_status'] or status['adb_status'] in ('offline', + 'missing'))) + should_restart_adb = should_restart_usb.union( + set(status['serial'] for status in statuses + if status['adb_status'] == 'unauthorized')) + should_reboot_device = should_restart_usb.union( + set(status['serial'] for status in statuses if status['denylisted'])) logger.debug('Should restart USB for:') for d in should_restart_usb: @@ -201,8 +196,8 @@ def RecoverDevices(devices, blacklist, enable_usb_reset=False): for d in should_reboot_device: logger.debug(' %s', d) - if blacklist: - blacklist.Reset() + if denylist: + denylist.Reset() if should_restart_adb: KillAllAdb() @@ -215,19 +210,19 @@ def RecoverDevices(devices, blacklist, enable_usb_reset=False): if enable_usb_reset: reset_usb.reset_android_usb(serial) else: - logger.warning('USB reset disabled for %s (crbug.com/642914)', - serial) + logger.warning('USB reset disabled for %s (crbug.com/642914)', serial) except IOError: logger.exception('Unable to reset USB for %s.', serial) - if blacklist: - blacklist.Extend([serial], reason='USB failure') + if denylist: + denylist.Extend([serial], reason='USB failure') except device_errors.DeviceUnreachableError: logger.exception('Unable to reset USB for %s.', serial) - if blacklist: - blacklist.Extend([serial], reason='offline') + if denylist: + denylist.Extend([serial], reason='offline') device_utils.DeviceUtils.parallel(devices).pMap( - RecoverDevice, blacklist, + RecoverDevice, + denylist, should_reboot=lambda device: device.serial in should_reboot_device) @@ -235,27 +230,35 @@ def main(): parser = argparse.ArgumentParser() logging_common.AddLoggingArguments(parser) script_common.AddEnvironmentArguments(parser) - parser.add_argument('--blacklist-file', help='Device blacklist JSON file.') - parser.add_argument('--known-devices-file', action='append', default=[], - dest='known_devices_files', - help='Path to known device lists.') - parser.add_argument('--enable-usb-reset', action='store_true', - help='Reset USB if necessary.') + # TODO(crbug.com/1097306): Remove this once callers switch to --denylist-file. + parser.add_argument('--blacklist-file', help=argparse.SUPPRESS) + parser.add_argument('--denylist-file', help='Device denylist JSON file.') + parser.add_argument( + '--known-devices-file', + action='append', + default=[], + dest='known_devices_files', + help='Path to known device lists.') + parser.add_argument( + '--enable-usb-reset', action='store_true', help='Reset USB if necessary.') args = parser.parse_args() logging_common.InitializeLogging(args) script_common.InitializeEnvironment(args) - blacklist = (device_blacklist.Blacklist(args.blacklist_file) - if args.blacklist_file - else None) + denylist = (device_denylist.Denylist(args.denylist_file) + if args.denylist_file else None) + # TODO(crbug.com/1097306): Remove this once callers switch to --denylist-file. + if not denylist and args.blacklist_file: + denylist = device_denylist.Denylist(args.blacklist_file) expected_devices = device_status.GetExpectedDevices(args.known_devices_files) usb_devices = set(lsusb.get_android_devices()) - devices = [device_utils.DeviceUtils(s) - for s in expected_devices.union(usb_devices)] + devices = [ + device_utils.DeviceUtils(s) for s in expected_devices.union(usb_devices) + ] - RecoverDevices(devices, blacklist, enable_usb_reset=args.enable_usb_reset) + RecoverDevices(devices, denylist, enable_usb_reset=args.enable_usb_reset) if __name__ == '__main__': diff --git a/catapult/devil/devil/android/tools/device_status.py b/catapult/devil/devil/android/tools/device_status.py index dbbf2908..d0eb7df1 100755 --- a/catapult/devil/devil/android/tools/device_status.py +++ b/catapult/devil/devil/android/tools/device_status.py @@ -2,7 +2,6 @@ # Copyright 2016 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A script to keep track of devices across builds and report state.""" import argparse @@ -14,10 +13,10 @@ import sys if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil.android import battery_utils -from devil.android import device_blacklist +from devil.android import device_denylist from devil.android import device_errors from devil.android import device_list from devil.android import device_utils @@ -32,11 +31,11 @@ logger = logging.getLogger(__name__) _RE_DEVICE_ID = re.compile(r'Device ID = (\d+)') -def IsBlacklisted(serial, blacklist): - return blacklist and serial in blacklist.Read() +def IsDenylisted(serial, denylist): + return denylist and serial in denylist.Read() -def _BatteryStatus(device, blacklist): +def _BatteryStatus(device, denylist): battery_info = {} try: battery = battery_utils.BatteryUtils(device) @@ -48,23 +47,22 @@ def _BatteryStatus(device, blacklist): battery = battery_utils.BatteryUtils(device) if not battery.GetCharging(): battery.SetCharging(True) - if blacklist: - blacklist.Extend([device.adb.GetDeviceSerial()], reason='low_battery') + if denylist: + denylist.Extend([device.adb.GetDeviceSerial()], reason='low_battery') except (device_errors.CommandFailedError, device_errors.DeviceUnreachableError): - logger.exception('Failed to get battery information for %s', - str(device)) + logger.exception('Failed to get battery information for %s', str(device)) return battery_info -def DeviceStatus(devices, blacklist): +def DeviceStatus(devices, denylist): """Generates status information for the given devices. Args: devices: The devices to generate status for. - blacklist: The current device blacklist. + denylist: The current device denylist. Returns: A dict of the following form: { @@ -72,8 +70,8 @@ def DeviceStatus(devices, blacklist): 'serial': '<serial>', 'adb_status': str, 'usb_status': bool, - 'blacklisted': bool, - # only if the device is connected and not blacklisted + 'denylisted': bool, + # only if the device is connected and not denylisted 'type': ro.build.product, 'build': ro.build.id, 'build_detail': ro.build.fingerprint, @@ -87,25 +85,25 @@ def DeviceStatus(devices, blacklist): } """ adb_devices = { - a[0].GetDeviceSerial(): a - for a in adb_wrapper.AdbWrapper.Devices(desired_state=None, long_list=True) + a[0].GetDeviceSerial(): a + for a in adb_wrapper.AdbWrapper.Devices( + desired_state=None, long_list=True) } usb_devices = set(lsusb.get_android_devices()) - def blacklisting_device_status(device): + def denylisting_device_status(device): serial = device.adb.GetDeviceSerial() - adb_status = ( - adb_devices[serial][1] if serial in adb_devices - else 'missing') + adb_status = (adb_devices[serial][1] + if serial in adb_devices else 'missing') usb_status = bool(serial in usb_devices) device_status = { - 'serial': serial, - 'adb_status': adb_status, - 'usb_status': usb_status, + 'serial': serial, + 'adb_status': adb_status, + 'usb_status': usb_status, } - if not IsBlacklisted(serial, blacklist): + if not IsDenylisted(serial, denylist): if adb_status == 'device': try: build_product = device.build_product @@ -113,50 +111,50 @@ def DeviceStatus(devices, blacklist): build_fingerprint = device.build_fingerprint build_description = device.build_description wifi_ip = device.GetProp('dhcp.wlan0.ipaddress') - battery_info = _BatteryStatus(device, blacklist) + battery_info = _BatteryStatus(device, denylist) try: imei_slice = device.GetIMEI() except device_errors.CommandFailedError: logging.exception('Unable to fetch IMEI for %s.', str(device)) imei_slice = 'unknown' - if (device.product_name == 'mantaray' and - battery_info.get('AC powered', None) != 'true'): + if (device.product_name == 'mantaray' + and battery_info.get('AC powered', None) != 'true'): logger.error('Mantaray device not connected to AC power.') device_status.update({ - 'ro.build.product': build_product, - 'ro.build.id': build_id, - 'ro.build.fingerprint': build_fingerprint, - 'ro.build.description': build_description, - 'battery': battery_info, - 'imei_slice': imei_slice, - 'wifi_ip': wifi_ip, + 'ro.build.product': build_product, + 'ro.build.id': build_id, + 'ro.build.fingerprint': build_fingerprint, + 'ro.build.description': build_description, + 'battery': battery_info, + 'imei_slice': imei_slice, + 'wifi_ip': wifi_ip, }) except (device_errors.CommandFailedError, device_errors.DeviceUnreachableError): logger.exception('Failure while getting device status for %s.', str(device)) - if blacklist: - blacklist.Extend([serial], reason='status_check_failure') + if denylist: + denylist.Extend([serial], reason='status_check_failure') except device_errors.CommandTimeoutError: logger.exception('Timeout while getting device status for %s.', str(device)) - if blacklist: - blacklist.Extend([serial], reason='status_check_timeout') + if denylist: + denylist.Extend([serial], reason='status_check_timeout') - elif blacklist: - blacklist.Extend([serial], - reason=adb_status if usb_status else 'offline') + elif denylist: + denylist.Extend([serial], + reason=adb_status if usb_status else 'offline') - device_status['blacklisted'] = IsBlacklisted(serial, blacklist) + device_status['denylisted'] = IsDenylisted(serial, denylist) return device_status parallel_devices = device_utils.DeviceUtils.parallel(devices) - statuses = parallel_devices.pMap(blacklisting_device_status).pGet(None) + statuses = parallel_devices.pMap(denylisting_device_status).pGet(None) return statuses @@ -165,12 +163,12 @@ def _LogStatuses(statuses): for status in statuses: logger.info(status['serial']) adb_status = status.get('adb_status') - blacklisted = status.get('blacklisted') + denylisted = status.get('denylisted') logger.info(' USB status: %s', 'online' if status.get('usb_status') else 'offline') logger.info(' ADB status: %s', adb_status) - logger.info(' Blacklisted: %s', str(blacklisted)) - if adb_status == 'device' and not blacklisted: + logger.info(' Denylisted: %s', str(denylisted)) + if adb_status == 'device' and not denylisted: logger.info(' Device type: %s', status.get('ro.build.product')) logger.info(' OS build: %s', status.get('ro.build.id')) logger.info(' OS build fingerprint: %s', @@ -189,25 +187,21 @@ def _WriteBuildbotFile(file_path, statuses): for status in statuses: try: if status['adb_status'] == 'device': - f.write('{serial} {adb_status} {build_product} {build_id} ' - '{temperature:.1f}C {level}%\n'.format( - serial=status['serial'], - adb_status=status['adb_status'], - build_product=status['type'], - build_id=status['build'], - temperature=float(status['battery']['temperature']) / 10, - level=status['battery']['level'] - )) + f.write( + '{serial} {adb_status} {build_product} {build_id} ' + '{temperature:.1f}C {level}%\n'.format( + serial=status['serial'], + adb_status=status['adb_status'], + build_product=status['type'], + build_id=status['build'], + temperature=float(status['battery']['temperature']) / 10, + level=status['battery']['level'])) elif status.get('usb_status', False): f.write('{serial} {adb_status}\n'.format( - serial=status['serial'], - adb_status=status['adb_status'] - )) + serial=status['serial'], adb_status=status['adb_status'])) else: - f.write('{serial} offline\n'.format( - serial=status['serial'] - )) - except Exception: # pylint: disable=broad-except + f.write('{serial} offline\n'.format(serial=status['serial'])) + except Exception: # pylint: disable=broad-except pass @@ -229,19 +223,32 @@ def GetExpectedDevices(known_devices_files): def AddArguments(parser): - parser.add_argument('--json-output', - help='Output JSON information into a specified file.') - parser.add_argument('--blacklist-file', help='Device blacklist JSON file.') - parser.add_argument('--known-devices-file', action='append', default=[], - dest='known_devices_files', - help='Path to known device lists.') - parser.add_argument('--buildbot-path', '-b', - default='/home/chrome-bot/.adb_device_info', - help='Absolute path to buildbot file location') - parser.add_argument('-w', '--overwrite-known-devices-files', - action='store_true', - help='If set, overwrites known devices files wiht new ' - 'values.') + parser.add_argument( + '--json-output', help='Output JSON information into a specified file.') + parser.add_argument('--denylist-file', help='Device denylist JSON file.') + # TODO(crbug.com/1097306): Remove this once //testing/scripts/host_info.py + # stops using it. + parser.add_argument('--blacklist-file', + dest='denylist_file', + help=argparse.SUPPRESS) + parser.add_argument( + '--known-devices-file', + action='append', + default=[], + dest='known_devices_files', + help='Path to known device lists.') + parser.add_argument( + '--buildbot-path', + '-b', + default='/home/chrome-bot/.adb_device_info', + help='Absolute path to buildbot file location') + parser.add_argument( + '-w', + '--overwrite-known-devices-files', + action='store_true', + help='If set, overwrites known devices files wiht new ' + 'values.') + def main(): parser = argparse.ArgumentParser() @@ -253,16 +260,16 @@ def main(): logging_common.InitializeLogging(args) script_common.InitializeEnvironment(args) - blacklist = (device_blacklist.Blacklist(args.blacklist_file) - if args.blacklist_file - else None) + denylist = (device_denylist.Denylist(args.denylist_file) + if args.denylist_file else None) expected_devices = GetExpectedDevices(args.known_devices_files) usb_devices = set(lsusb.get_android_devices()) - devices = [device_utils.DeviceUtils(s) - for s in expected_devices.union(usb_devices)] + devices = [ + device_utils.DeviceUtils(s) for s in expected_devices.union(usb_devices) + ] - statuses = DeviceStatus(devices, blacklist) + statuses = DeviceStatus(devices, denylist) # Log the state of all devices. _LogStatuses(statuses) @@ -279,12 +286,15 @@ def main(): # Dump the device statuses to JSON. if args.json_output: with open(args.json_output, 'wb') as f: - f.write(json.dumps( - statuses, indent=4, sort_keys=True, separators=(',', ': '))) - - live_devices = [status['serial'] for status in statuses - if (status['adb_status'] == 'device' - and not IsBlacklisted(status['serial'], blacklist))] + f.write( + json.dumps( + statuses, indent=4, sort_keys=True, separators=(',', ': '))) + + live_devices = [ + status['serial'] for status in statuses + if (status['adb_status'] == 'device' + and not IsDenylisted(status['serial'], denylist)) + ] # If all devices failed, or if there are no devices, it's an infra error. if not live_devices: diff --git a/catapult/devil/devil/android/tools/flash_device.py b/catapult/devil/devil/android/tools/flash_device.py index 8b51c604..cb06a120 100755 --- a/catapult/devil/devil/android/tools/flash_device.py +++ b/catapult/devil/devil/android/tools/flash_device.py @@ -9,14 +9,17 @@ import os import sys if __name__ == '__main__': - sys.path.append(os.path.abspath(os.path.join( - os.path.dirname(__file__), '..', '..', '..'))) -from devil.android import device_blacklist -from devil.android import device_utils + sys.path.append( + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) +from devil.android import device_denylist +from devil.android import device_errors from devil.android import fastboot_utils +from devil.android.sdk import fastboot from devil.android.tools import script_common from devil.constants import exit_codes from devil.utils import logging_common +from devil.utils import parallelizer logger = logging.getLogger(__name__) @@ -24,18 +27,18 @@ logger = logging.getLogger(__name__) def main(): parser = argparse.ArgumentParser() parser.add_argument('build_path', help='Path to android build.') - parser.add_argument('-w', '--wipe', action='store_true', - help='If set, wipes user data') + parser.add_argument( + '-w', '--wipe', action='store_true', help='If set, wipes user data') logging_common.AddLoggingArguments(parser) script_common.AddDeviceArguments(parser) args = parser.parse_args() logging_common.InitializeLogging(args) - if args.blacklist_file: - blacklist = device_blacklist.Blacklist(args.blacklist_file).Read() - if blacklist: - logger.critical('Device(s) in blacklist, not flashing devices:') - for key in blacklist: + if args.denylist_file: + denylist = device_denylist.Denylist(args.denylist_file).Read() + if denylist: + logger.critical('Device(s) in denylist, not flashing devices:') + for key in denylist: logger.critical(' %s', key) return exit_codes.INFRA @@ -43,16 +46,30 @@ def main(): failed_devices = [] def flash(device): - fastboot = fastboot_utils.FastbootUtils(device) try: - fastboot.FlashDevice(args.build_path, wipe=args.wipe) + device.FlashDevice(args.build_path, wipe=args.wipe) flashed_devices.append(device) except Exception: # pylint: disable=broad-except logger.exception('Device %s failed to flash.', str(device)) failed_devices.append(device) - devices = script_common.GetDevices(args.devices, args.blacklist_file) - device_utils.DeviceUtils.parallel(devices).pMap(flash) + devices = [] + try: + adb_devices = script_common.GetDevices(args.devices, args.denylist_file) + devices += [fastboot_utils.FastbootUtils(device=d) for d in adb_devices] + except device_errors.NoDevicesError: + # Don't bail out if we're not looking for any particular device and there's + # at least one sitting in fastboot mode. Note that if we ARE looking for a + # particular device, and it's in fastboot mode, this will still fail. + fastboot_devices = fastboot.Fastboot.Devices() + if args.devices or not fastboot_devices: + raise + devices += [ + fastboot_utils.FastbootUtils(fastbooter=d) for d in fastboot_devices + ] + + parallel_devices = parallelizer.SyncParallelizer(devices) + parallel_devices.pMap(flash) if flashed_devices: logger.info('The following devices were flashed:') @@ -63,5 +80,6 @@ def main(): return exit_codes.INFRA return 0 + if __name__ == '__main__': sys.exit(main()) diff --git a/catapult/devil/devil/android/tools/keyboard.py b/catapult/devil/devil/android/tools/keyboard.py index c5cb6149..e400bca4 100755 --- a/catapult/devil/devil/android/tools/keyboard.py +++ b/catapult/devil/devil/android/tools/keyboard.py @@ -2,7 +2,6 @@ # Copyright 2016 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Use your keyboard as your phone's keyboard. Experimental.""" import argparse @@ -14,56 +13,55 @@ import tty if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil import base_error from devil.android.sdk import keyevent from devil.android.tools import script_common from devil.utils import logging_common - _KEY_MAPPING = { - '\x08': keyevent.KEYCODE_DEL, - '\x0a': keyevent.KEYCODE_ENTER, - ' ': keyevent.KEYCODE_SPACE, - '.': keyevent.KEYCODE_PERIOD, - '0': keyevent.KEYCODE_0, - '1': keyevent.KEYCODE_1, - '2': keyevent.KEYCODE_2, - '3': keyevent.KEYCODE_3, - '4': keyevent.KEYCODE_4, - '5': keyevent.KEYCODE_5, - '6': keyevent.KEYCODE_6, - '7': keyevent.KEYCODE_7, - '8': keyevent.KEYCODE_8, - '9': keyevent.KEYCODE_9, - 'a': keyevent.KEYCODE_A, - 'b': keyevent.KEYCODE_B, - 'c': keyevent.KEYCODE_C, - 'd': keyevent.KEYCODE_D, - 'e': keyevent.KEYCODE_E, - 'f': keyevent.KEYCODE_F, - 'g': keyevent.KEYCODE_G, - 'h': keyevent.KEYCODE_H, - 'i': keyevent.KEYCODE_I, - 'j': keyevent.KEYCODE_J, - 'k': keyevent.KEYCODE_K, - 'l': keyevent.KEYCODE_L, - 'm': keyevent.KEYCODE_M, - 'n': keyevent.KEYCODE_N, - 'o': keyevent.KEYCODE_O, - 'p': keyevent.KEYCODE_P, - 'q': keyevent.KEYCODE_Q, - 'r': keyevent.KEYCODE_R, - 's': keyevent.KEYCODE_S, - 't': keyevent.KEYCODE_T, - 'u': keyevent.KEYCODE_U, - 'v': keyevent.KEYCODE_V, - 'w': keyevent.KEYCODE_W, - 'x': keyevent.KEYCODE_X, - 'y': keyevent.KEYCODE_Y, - 'z': keyevent.KEYCODE_Z, - '\x7f': keyevent.KEYCODE_DEL, + '\x08': keyevent.KEYCODE_DEL, + '\x0a': keyevent.KEYCODE_ENTER, + ' ': keyevent.KEYCODE_SPACE, + '.': keyevent.KEYCODE_PERIOD, + '0': keyevent.KEYCODE_0, + '1': keyevent.KEYCODE_1, + '2': keyevent.KEYCODE_2, + '3': keyevent.KEYCODE_3, + '4': keyevent.KEYCODE_4, + '5': keyevent.KEYCODE_5, + '6': keyevent.KEYCODE_6, + '7': keyevent.KEYCODE_7, + '8': keyevent.KEYCODE_8, + '9': keyevent.KEYCODE_9, + 'a': keyevent.KEYCODE_A, + 'b': keyevent.KEYCODE_B, + 'c': keyevent.KEYCODE_C, + 'd': keyevent.KEYCODE_D, + 'e': keyevent.KEYCODE_E, + 'f': keyevent.KEYCODE_F, + 'g': keyevent.KEYCODE_G, + 'h': keyevent.KEYCODE_H, + 'i': keyevent.KEYCODE_I, + 'j': keyevent.KEYCODE_J, + 'k': keyevent.KEYCODE_K, + 'l': keyevent.KEYCODE_L, + 'm': keyevent.KEYCODE_M, + 'n': keyevent.KEYCODE_N, + 'o': keyevent.KEYCODE_O, + 'p': keyevent.KEYCODE_P, + 'q': keyevent.KEYCODE_Q, + 'r': keyevent.KEYCODE_R, + 's': keyevent.KEYCODE_S, + 't': keyevent.KEYCODE_T, + 'u': keyevent.KEYCODE_U, + 'v': keyevent.KEYCODE_V, + 'w': keyevent.KEYCODE_W, + 'x': keyevent.KEYCODE_X, + 'y': keyevent.KEYCODE_Y, + 'z': keyevent.KEYCODE_Z, + '\x7f': keyevent.KEYCODE_DEL, } diff --git a/catapult/devil/devil/android/tools/provision_devices.py b/catapult/devil/devil/android/tools/provision_devices.py index 47b1dc3f..4b0f44a2 100755 --- a/catapult/devil/devil/android/tools/provision_devices.py +++ b/catapult/devil/devil/android/tools/provision_devices.py @@ -3,7 +3,6 @@ # Copyright (c) 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Provisions Android devices with settings required for bots. Usage: @@ -27,11 +26,11 @@ import _strptime # pylint: disable=unused-import if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil.android import battery_utils -from devil.android import device_blacklist +from devil.android import device_denylist from devil.android import device_errors from devil.android import device_temp_file from devil.android import device_utils @@ -53,7 +52,7 @@ _SYSTEM_WEBVIEW_NAMES = ['webview', 'WebViewGoogle'] _CHROME_PACKAGE_REGEX = re.compile('.*chrom.*') _TOMBSTONE_REGEX = re.compile('tombstone.*') _STANDALONE_VR_DEVICES = [ - 'vega', # Lenovo Mirage Solo + 'vega', # Lenovo Mirage Solo ] @@ -66,53 +65,51 @@ class _DEFAULT_TIMEOUTS(object): class ProvisionStep(object): - def __init__(self, cmd, reboot=False): self.cmd = cmd self.reboot = reboot -def ProvisionDevices( - devices, - blacklist_file, - adb_key_files=None, - disable_location=False, - disable_mock_location=False, - disable_network=False, - disable_system_chrome=False, - emulators=False, - enable_java_debug=False, - max_battery_temp=None, - min_battery_level=None, - output_device_blacklist=None, - reboot_timeout=None, - remove_system_webview=False, - system_app_remove_list=None, - system_package_remove_list=None, - wipe=True): - blacklist = (device_blacklist.Blacklist(blacklist_file) - if blacklist_file - else None) +def ProvisionDevices(devices, + denylist_file, + adb_key_files=None, + disable_location=False, + disable_mock_location=False, + disable_network=False, + disable_system_chrome=False, + emulators=False, + enable_java_debug=False, + max_battery_temp=None, + min_battery_level=None, + output_device_denylist=None, + reboot_timeout=None, + remove_system_webview=False, + system_app_remove_list=None, + system_package_remove_list=None, + wipe=True): + denylist = (device_denylist.Denylist(denylist_file) + if denylist_file else None) system_app_remove_list = system_app_remove_list or [] system_package_remove_list = system_package_remove_list or [] try: - devices = script_common.GetDevices(devices, blacklist) + devices = script_common.GetDevices(devices, denylist) except device_errors.NoDevicesError: logging.error('No available devices to provision.') - if blacklist: - logging.error('Local device blacklist: %s', blacklist.Read()) + if denylist: + logging.error('Local device denylist: %s', denylist.Read()) raise - devices = [d for d in devices - if not emulators or d.adb.is_emulator] + devices = [d for d in devices if not emulators or d.adb.is_emulator] parallel_devices = device_utils.DeviceUtils.parallel(devices) steps = [] if wipe: steps += [ProvisionStep(lambda d: Wipe(d, adb_key_files), reboot=True)] - steps += [ProvisionStep( - lambda d: SetProperties(d, enable_java_debug, disable_location, - disable_mock_location), - reboot=not emulators)] + steps += [ + ProvisionStep( + lambda d: SetProperties(d, enable_java_debug, disable_location, + disable_mock_location), + reboot=not emulators) + ] if disable_network: steps.append(ProvisionStep(DisableNetwork)) @@ -121,37 +118,36 @@ def ProvisionDevices( steps.append(ProvisionStep(DisableSystemChrome)) if max_battery_temp: - steps.append(ProvisionStep( - lambda d: WaitForBatteryTemperature(d, max_battery_temp))) + steps.append( + ProvisionStep(lambda d: WaitForBatteryTemperature(d, max_battery_temp))) if min_battery_level: - steps.append(ProvisionStep( - lambda d: WaitForCharge(d, min_battery_level))) + steps.append(ProvisionStep(lambda d: WaitForCharge(d, min_battery_level))) if remove_system_webview: system_app_remove_list.extend(_SYSTEM_WEBVIEW_NAMES) if system_app_remove_list or system_package_remove_list: - steps.append(ProvisionStep( - lambda d: RemoveSystemApps( - d, system_app_remove_list, system_package_remove_list))) + steps.append( + ProvisionStep(lambda d: RemoveSystemApps(d, system_app_remove_list, + system_package_remove_list))) steps.append(ProvisionStep(SetDate)) steps.append(ProvisionStep(CheckExternalStorage)) steps.append(ProvisionStep(StandaloneVrDeviceSetup)) - parallel_devices.pMap(ProvisionDevice, steps, blacklist, reboot_timeout) + parallel_devices.pMap(ProvisionDevice, steps, denylist, reboot_timeout) - blacklisted_devices = blacklist.Read() if blacklist else [] - if output_device_blacklist: - with open(output_device_blacklist, 'w') as f: - json.dump(blacklisted_devices, f) - if all(d in blacklisted_devices for d in devices): + denylisted_devices = denylist.Read() if denylist else [] + if output_device_denylist: + with open(output_device_denylist, 'w') as f: + json.dump(denylisted_devices, f) + if all(d in denylisted_devices for d in devices): raise device_errors.NoDevicesError return 0 -def ProvisionDevice(device, steps, blacklist, reboot_timeout=None): +def ProvisionDevice(device, steps, denylist, reboot_timeout=None): try: if not reboot_timeout: if device.build_version_sdk >= version_codes.LOLLIPOP: @@ -171,27 +167,27 @@ def ProvisionDevice(device, steps, blacklist, reboot_timeout=None): device.adb.WaitForDevice() except device_errors.CommandTimeoutError: - logger.exception('Timed out waiting for device %s. Adding to blacklist.', + logger.exception('Timed out waiting for device %s. Adding to denylist.', str(device)) - if blacklist: - blacklist.Extend([str(device)], reason='provision_timeout') + if denylist: + denylist.Extend([str(device)], reason='provision_timeout') except (device_errors.CommandFailedError, device_errors.DeviceUnreachableError): - logger.exception('Failed to provision device %s. Adding to blacklist.', + logger.exception('Failed to provision device %s. Adding to denylist.', str(device)) - if blacklist: - blacklist.Extend([str(device)], reason='provision_failure') + if denylist: + denylist.Extend([str(device)], reason='provision_failure') def Wipe(device, adb_key_files=None): - if (device.IsUserBuild() or - device.build_version_sdk >= version_codes.MARSHMALLOW): + if (device.IsUserBuild() + or device.build_version_sdk >= version_codes.MARSHMALLOW): WipeChromeData(device) package = 'com.google.android.gms' - if device.GetApplicationPaths(package): - version_name = device.GetApplicationVersion(package) + version_name = device.GetApplicationVersion(package) + if version_name: logger.info('Version name for %s is %s', package, version_name) else: logger.info('Package %s is not installed', package) @@ -219,10 +215,12 @@ def WipeChromeData(device): try: if device.IsUserBuild(): _UninstallIfMatch(device, _CHROME_PACKAGE_REGEX) - device.RunShellCommand('rm -rf %s/*' % device.GetExternalStoragePath(), - shell=True, check_return=True) - device.RunShellCommand('rm -rf /data/local/tmp/*', - shell=True, check_return=True) + device.RunShellCommand( + 'rm -rf %s/*' % device.GetExternalStoragePath(), + shell=True, + check_return=True) + device.RunShellCommand( + 'rm -rf /data/local/tmp/*', shell=True, check_return=True) else: device.EnableRoot() _UninstallIfMatch(device, _CHROME_PACKAGE_REGEX) @@ -233,19 +231,23 @@ def WipeChromeData(device): _WipeFileOrDir(device, '/data/local/chrome-command-line') _WipeFileOrDir(device, '/data/local/.config/') _WipeFileOrDir(device, '/data/local/tmp/') - device.RunShellCommand('rm -rf %s/*' % device.GetExternalStoragePath(), - shell=True, check_return=True) + device.RunShellCommand( + 'rm -rf %s/*' % device.GetExternalStoragePath(), + shell=True, + check_return=True) except device_errors.CommandFailedError: logger.exception('Possible failure while wiping the device. ' 'Attempting to continue.') def _UninstallIfMatch(device, pattern): - installed_packages = device.RunShellCommand( - ['pm', 'list', 'packages'], check_return=True) + installed_packages = device.RunShellCommand(['pm', 'list', 'packages'], + check_return=True) installed_system_packages = [ - pkg.split(':')[1] for pkg in device.RunShellCommand( - ['pm', 'list', 'packages', '-s'], check_return=True)] + pkg.split(':')[1] + for pkg in device.RunShellCommand(['pm', 'list', 'packages', '-s'], + check_return=True) + ] for package_output in installed_packages: package = package_output.split(":")[1] if pattern.match(package) and package not in installed_system_packages: @@ -279,10 +281,9 @@ def WipeDevice(device, adb_key_files): device.EnableRoot() device_authorized = device.FileExists(adb_wrapper.ADB_KEYS_FILE) if device_authorized: - adb_keys = device.ReadFile(adb_wrapper.ADB_KEYS_FILE, - as_root=True).splitlines() - device.RunShellCommand(['wipe', 'data'], - as_root=True, check_return=True) + adb_keys = device.ReadFile( + adb_wrapper.ADB_KEYS_FILE, as_root=True).splitlines() + device.RunShellCommand(['wipe', 'data'], as_root=True, check_return=True) device.adb.WaitForDevice() if device_authorized: @@ -303,12 +304,15 @@ def WipeDevice(device, adb_key_files): def _WriteAdbKeysFile(device, adb_keys_string): dir_path = posixpath.dirname(adb_wrapper.ADB_KEYS_FILE) device.RunShellCommand(['mkdir', '-p', dir_path], - as_root=True, check_return=True) + as_root=True, + check_return=True) device.RunShellCommand(['restorecon', dir_path], - as_root=True, check_return=True) + as_root=True, + check_return=True) device.WriteFile(adb_wrapper.ADB_KEYS_FILE, adb_keys_string, as_root=True) device.RunShellCommand(['restorecon', adb_wrapper.ADB_KEYS_FILE], - as_root=True, check_return=True) + as_root=True, + check_return=True) def SetProperties(device, enable_java_debug, disable_location, @@ -322,21 +326,20 @@ def SetProperties(device, enable_java_debug, disable_location, _ConfigureLocalProperties(device, enable_java_debug) else: logger.warning('Cannot configure properties in user builds.') - settings.ConfigureContentSettings( - device, settings.DETERMINISTIC_DEVICE_SETTINGS) + settings.ConfigureContentSettings(device, + settings.DETERMINISTIC_DEVICE_SETTINGS) if disable_location: - settings.ConfigureContentSettings( - device, settings.DISABLE_LOCATION_SETTINGS) + settings.ConfigureContentSettings(device, + settings.DISABLE_LOCATION_SETTINGS) else: - settings.ConfigureContentSettings( - device, settings.ENABLE_LOCATION_SETTINGS) + settings.ConfigureContentSettings(device, settings.ENABLE_LOCATION_SETTINGS) if disable_mock_location: - settings.ConfigureContentSettings( - device, settings.DISABLE_MOCK_LOCATION_SETTINGS) + settings.ConfigureContentSettings(device, + settings.DISABLE_MOCK_LOCATION_SETTINGS) else: - settings.ConfigureContentSettings( - device, settings.ENABLE_MOCK_LOCATION_SETTINGS) + settings.ConfigureContentSettings(device, + settings.ENABLE_MOCK_LOCATION_SETTINGS) settings.SetLockScreenSettings(device) @@ -345,18 +348,19 @@ def SetProperties(device, enable_java_debug, disable_location, def DisableNetwork(device): - settings.ConfigureContentSettings( - device, settings.NETWORK_DISABLED_SETTINGS) + settings.ConfigureContentSettings(device, settings.NETWORK_DISABLED_SETTINGS) if device.build_version_sdk >= version_codes.MARSHMALLOW: # Ensure that NFC is also switched off. device.RunShellCommand(['svc', 'nfc', 'disable'], - as_root=True, check_return=True) + as_root=True, + check_return=True) def DisableSystemChrome(device): # The system chrome version on the device interferes with some tests. device.RunShellCommand(['pm', 'disable', 'com.android.chrome'], - as_root=True, check_return=True) + as_root=True, + check_return=True) def _FindSystemPackagePaths(device, system_package_list): @@ -376,8 +380,8 @@ def _FindSystemAppPaths(device, system_app_list): return found_paths -def RemoveSystemApps( - device, system_app_remove_list, system_package_remove_list): +def RemoveSystemApps(device, system_app_remove_list, + system_package_remove_list): """Attempts to remove the provided system apps from the given device. Arguments: @@ -418,19 +422,17 @@ def _ConfigureLocalProperties(device, java_debug=True): 'ro.test_harness=1', 'ro.audio.silent=1', 'ro.setupwizard.mode=DISABLED', - ] + ] if java_debug: - local_props.append( - '%s=all' % device_utils.DeviceUtils.JAVA_ASSERT_PROPERTY) + local_props.append('%s=all' % device_utils.DeviceUtils.JAVA_ASSERT_PROPERTY) local_props.append('debug.checkjni=1') try: device.WriteFile( - device.LOCAL_PROPERTIES_PATH, - '\n'.join(local_props), as_root=True) + device.LOCAL_PROPERTIES_PATH, '\n'.join(local_props), as_root=True) # Android will not respect the local props file if it is world writable. - device.RunShellCommand( - ['chmod', '644', device.LOCAL_PROPERTIES_PATH], - as_root=True, check_return=True) + device.RunShellCommand(['chmod', '644', device.LOCAL_PROPERTIES_PATH], + as_root=True, + check_return=True) except device_errors.CommandFailedError: logger.exception('Failed to configure local properties.') @@ -478,8 +480,8 @@ def SetDate(device): get_date_command.append('+"%Y%m%d.%H%M%S"') device_time = device.RunShellCommand( - get_date_command, check_return=True, - as_root=True, single_line=True).replace('"', '') + get_date_command, check_return=True, as_root=True, + single_line=True).replace('"', '') device_time = datetime.datetime.strptime(device_time, "%Y%m%d.%H%M%S") correct_time = datetime.datetime.strptime(strgmtime, date_format) tdelta = (correct_time - device_time).seconds @@ -504,8 +506,7 @@ def SetDate(device): # The following intent can take a bit to complete when ran shortly after # device boot-up. device.BroadcastIntent( - intent.Intent(action='android.intent.action.TIME_SET'), - timeout=180) + intent.Intent(action='android.intent.action.TIME_SET'), timeout=180) def LogDeviceProperties(device): @@ -529,7 +530,8 @@ def CheckExternalStorage(device): except device_errors.CommandFailedError: logger.info('External storage not writable. Remounting / as RW') device.RunShellCommand(['mount', '-o', 'remount,rw', '/'], - check_return=True, as_root=True) + check_return=True, + as_root=True) device.EnableRoot() with device_temp_file.DeviceTempFile( device.adb, suffix='.sh', dir=device.GetExternalStoragePath()) as f: @@ -546,8 +548,11 @@ def StandaloneVrDeviceSetup(device): return # Modify VrCore's settings so that any first time setup, etc. is skipped. - shared_pref = shared_prefs.SharedPrefs(device, 'com.google.vr.vrcore', - 'VrCoreSettings.xml', use_encrypted_path=True) + shared_pref = shared_prefs.SharedPrefs( + device, + 'com.google.vr.vrcore', + 'VrCoreSettings.xml', + use_encrypted_path=True) shared_pref.Load() # Skip first time setup. shared_pref.SetBoolean('DaydreamSetupComplete', True) @@ -576,84 +581,107 @@ def main(raw_args): script_common.AddDeviceArguments(parser) script_common.AddEnvironmentArguments(parser) parser.add_argument( - '--adb-key-files', type=str, nargs='+', + '--adb-key-files', + type=str, + nargs='+', help='list of adb keys to push to device') parser.add_argument( - '--disable-location', action='store_true', + '--disable-location', + action='store_true', help='disable Google location services on devices') parser.add_argument( - '--disable-mock-location', action='store_true', default=False, + '--disable-mock-location', + action='store_true', + default=False, help='Set ALLOW_MOCK_LOCATION to false') parser.add_argument( - '--disable-network', action='store_true', + '--disable-network', + action='store_true', help='disable network access on devices') parser.add_argument( - '--disable-java-debug', action='store_false', - dest='enable_java_debug', default=True, + '--disable-java-debug', + action='store_false', + dest='enable_java_debug', + default=True, help='disable Java property asserts and JNI checking') parser.add_argument( - '--disable-system-chrome', action='store_true', + '--disable-system-chrome', + action='store_true', help='DEPRECATED: use --remove-system-packages com.android.google ' - 'Disable the system chrome from devices.') + 'Disable the system chrome from devices.') parser.add_argument( - '--emulators', action='store_true', + '--emulators', + action='store_true', help='provision only emulators and ignore usb devices ' - '(this will not wipe emulators)') + '(this will not wipe emulators)') parser.add_argument( - '--max-battery-temp', type=int, metavar='NUM', + '--max-battery-temp', + type=int, + metavar='NUM', help='Wait for the battery to have this temp or lower.') parser.add_argument( - '--min-battery-level', type=int, metavar='NUM', + '--min-battery-level', + type=int, + metavar='NUM', help='wait for the device to reach this minimum battery' - ' level before trying to continue') - parser.add_argument( - '--output-device-blacklist', - help='Json file to output the device blacklist.') + ' level before trying to continue') + # TODO(crbug.com/1097306): Remove after updating clients. + parser.add_argument('--output-device-blacklist', + help=argparse.SUPPRESS) + parser.add_argument('--output-device-denylist', + help='Json file to output the device denylist.') parser.add_argument( - '--reboot-timeout', metavar='SECS', type=int, + '--reboot-timeout', + metavar='SECS', + type=int, help='when wiping the device, max number of seconds to' - ' wait after each reboot ' - '(default: %s)' % _DEFAULT_TIMEOUTS.HELP_TEXT) + ' wait after each reboot ' + '(default: %s)' % _DEFAULT_TIMEOUTS.HELP_TEXT) parser.add_argument( - '--remove-system-apps', nargs='*', dest='system_app_remove_list', + '--remove-system-apps', + nargs='*', + dest='system_app_remove_list', help='DEPRECATED: use --remove-system-packages instead. ' - 'The names of system apps to remove. ') + 'The names of system apps to remove. ') parser.add_argument( - '--remove-system-packages', nargs='*', dest='system_package_remove_list', + '--remove-system-packages', + nargs='*', + dest='system_package_remove_list', help='The names of system packages to remove.') parser.add_argument( - '--remove-system-webview', action='store_true', + '--remove-system-webview', + action='store_true', help='DEPRECATED: use --remove-system-packages ' - 'com.google.android.webview com.android.webview ' - 'Remove the system webview from devices.') + 'com.google.android.webview com.android.webview ' + 'Remove the system webview from devices.') parser.add_argument( - '--skip-wipe', action='store_true', default=False, + '--skip-wipe', + action='store_true', + default=False, help='do not wipe device data during provisioning') # No-op arguments for compatibility with build/android/provision_devices.py. # TODO(jbudorick): Remove these once all callers have stopped using them. parser.add_argument( - '--chrome-specific-wipe', action='store_true', - help=argparse.SUPPRESS) + '--chrome-specific-wipe', action='store_true', help=argparse.SUPPRESS) + parser.add_argument('--phase', action='append', help=argparse.SUPPRESS) parser.add_argument( - '--phase', action='append', - help=argparse.SUPPRESS) - parser.add_argument( - '-r', '--auto-reconnect', action='store_true', - help=argparse.SUPPRESS) - parser.add_argument( - '-t', '--target', - help=argparse.SUPPRESS) + '-r', '--auto-reconnect', action='store_true', help=argparse.SUPPRESS) + parser.add_argument('-t', '--target', help=argparse.SUPPRESS) args = parser.parse_args(raw_args) logging_common.InitializeLogging(args) script_common.InitializeEnvironment(args) + output_device_denylist = args.output_device_denylist + if not output_device_denylist and args.output_device_blacklist: + output_device_denylist = args.output_device_blacklist + try: return ProvisionDevices( args.devices, - args.blacklist_file, + args.denylist_file, adb_key_files=args.adb_key_files, disable_location=args.disable_location, disable_mock_location=args.disable_mock_location, @@ -663,7 +691,7 @@ def main(raw_args): enable_java_debug=args.enable_java_debug, max_battery_temp=args.max_battery_temp, min_battery_level=args.min_battery_level, - output_device_blacklist=args.output_device_blacklist, + output_device_denylist=output_device_denylist, reboot_timeout=args.reboot_timeout, remove_system_webview=args.remove_system_webview, system_app_remove_list=args.system_app_remove_list, diff --git a/catapult/devil/devil/android/tools/screenshot.py b/catapult/devil/devil/android/tools/screenshot.py index 3b3335c2..fe6be04d 100755 --- a/catapult/devil/devil/android/tools/screenshot.py +++ b/catapult/devil/devil/android/tools/screenshot.py @@ -2,7 +2,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Takes a screenshot from an Android device.""" import argparse @@ -11,8 +10,9 @@ import os import sys if __name__ == '__main__': - sys.path.append(os.path.abspath(os.path.join( - os.path.dirname(__file__), '..', '..', '..'))) + sys.path.append( + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil.android import device_utils from devil.android.tools import script_common from devil.utils import logging_common @@ -25,17 +25,22 @@ def main(): parser = argparse.ArgumentParser(description=__doc__) logging_common.AddLoggingArguments(parser) script_common.AddDeviceArguments(parser) - parser.add_argument('-f', '--file', metavar='FILE', - help='Save result to file instead of generating a ' - 'timestamped file name.') - parser.add_argument('host_file', nargs='?', - help='File to which the screenshot will be saved.') + parser.add_argument( + '-f', + '--file', + metavar='FILE', + help='Save result to file instead of generating a ' + 'timestamped file name.') + parser.add_argument( + 'host_file', + nargs='?', + help='File to which the screenshot will be saved.') args = parser.parse_args() host_file = args.host_file or args.file logging_common.InitializeLogging(args) - devices = script_common.GetDevices(args.devices, args.blacklist_file) + devices = script_common.GetDevices(args.devices, args.denylist_file) def screenshot(device): f = None @@ -43,8 +48,8 @@ def main(): root, ext = os.path.splitext(host_file) f = '%s_%s%s' % (root, str(device), ext) f = device.TakeScreenshot(f) - print 'Screenshot for device %s written to %s' % ( - str(device), os.path.abspath(f)) + print 'Screenshot for device %s written to %s' % (str(device), + os.path.abspath(f)) device_utils.DeviceUtils.parallel(devices).pMap(screenshot) return 0 diff --git a/catapult/devil/devil/android/tools/script_common.py b/catapult/devil/devil/android/tools/script_common.py index 897659bb..ae2b7a84 100644 --- a/catapult/devil/devil/android/tools/script_common.py +++ b/catapult/devil/devil/android/tools/script_common.py @@ -2,10 +2,11 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. +import argparse import os from devil import devil_env -from devil.android import device_blacklist +from devil.android import device_denylist from devil.android import device_errors from devil.android import device_utils @@ -20,8 +21,7 @@ def AddEnvironmentArguments(parser): parser: an instance of argparse.ArgumentParser """ parser.add_argument( - '--adb-path', type=os.path.realpath, - help='Path to the adb binary') + '--adb-path', type=os.path.realpath, help='Path to the adb binary') def InitializeEnvironment(args): @@ -46,32 +46,51 @@ def InitializeEnvironment(args): devil_dynamic_config = devil_env.EmptyConfig() if args.adb_path: devil_dynamic_config['dependencies'].update( - devil_env.LocalConfigItem( - 'adb', devil_env.GetPlatform(), args.adb_path)) + devil_env.LocalConfigItem('adb', devil_env.GetPlatform(), + args.adb_path)) devil_env.config.Initialize(configs=[devil_dynamic_config]) def AddDeviceArguments(parser): - """Adds device and blacklist arguments to the provided parser. + """Adds device and denylist arguments to the provided parser. Args: parser: an instance of argparse.ArgumentParser """ parser.add_argument( - '-d', '--device', dest='devices', action='append', + '-d', + '--device', + dest='devices', + action='append', + default=[], help='Serial number of the Android device to use. (default: use all)') - parser.add_argument('--blacklist-file', help='Device blacklist JSON file.') + # TODO(crbug.com/1097306): Simplify this to an ungrouped --denylist-file + # once all uses of --blacklist-file / args.blacklist_file have been updated. + class DenylistAction(argparse.Action): -def GetDevices(requested_devices, blacklist_file): + #override + def __call__(self, parser, namespace, values, option_string=None): + setattr(namespace, 'denylist_file', values) + setattr(namespace, 'blacklist_file', values) + + denylist_group = parser.add_mutually_exclusive_group() + denylist_group.add_argument('--denylist-file', + help='Device denylist JSON file.', + action=DenylistAction) + denylist_group.add_argument('--blacklist-file', + help=argparse.SUPPRESS, + action=DenylistAction) + + +def GetDevices(requested_devices, denylist_file): """Gets a list of healthy devices matching the given parameters.""" - if not isinstance(blacklist_file, device_blacklist.Blacklist): - blacklist_file = (device_blacklist.Blacklist(blacklist_file) - if blacklist_file - else None) + if not isinstance(denylist_file, device_denylist.Denylist): + denylist_file = (device_denylist.Denylist(denylist_file) + if denylist_file else None) - devices = device_utils.DeviceUtils.HealthyDevices(blacklist_file) + devices = device_utils.DeviceUtils.HealthyDevices(denylist_file) if not devices: raise device_errors.NoDevicesError() elif requested_devices: @@ -80,8 +99,7 @@ def GetDevices(requested_devices, blacklist_file): missing = requested.difference(available) if missing: raise device_errors.DeviceUnreachableError(next(iter(missing))) - return sorted(device_utils.DeviceUtils(d) - for d in available.intersection(requested)) + return sorted( + device_utils.DeviceUtils(d) for d in available.intersection(requested)) else: return devices - diff --git a/catapult/devil/devil/android/tools/script_common_test.py b/catapult/devil/devil/android/tools/script_common_test.py index 30f9aaf9..f8269cc0 100755 --- a/catapult/devil/devil/android/tools/script_common_test.py +++ b/catapult/devil/devil/android/tools/script_common_test.py @@ -3,7 +3,6 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - import argparse import sys import tempfile @@ -23,44 +22,43 @@ with devil_env.SysPath(devil_env.DEPENDENCY_MANAGER_PATH): class GetDevicesTest(unittest.TestCase): - def testNoSpecs(self): devices = [ device_utils.DeviceUtils('123'), device_utils.DeviceUtils('456'), ] - with mock.patch('devil.android.device_utils.DeviceUtils.HealthyDevices', - return_value=devices): - self.assertEquals( - devices, - script_common.GetDevices(None, None)) + with mock.patch( + 'devil.android.device_utils.DeviceUtils.HealthyDevices', + return_value=devices): + self.assertEquals(devices, script_common.GetDevices(None, None)) def testWithDevices(self): devices = [ device_utils.DeviceUtils('123'), device_utils.DeviceUtils('456'), ] - with mock.patch('devil.android.device_utils.DeviceUtils.HealthyDevices', - return_value=devices): - self.assertEquals( - [device_utils.DeviceUtils('456')], - script_common.GetDevices(['456'], None)) + with mock.patch( + 'devil.android.device_utils.DeviceUtils.HealthyDevices', + return_value=devices): + self.assertEquals([device_utils.DeviceUtils('456')], + script_common.GetDevices(['456'], None)) def testMissingDevice(self): - with mock.patch('devil.android.device_utils.DeviceUtils.HealthyDevices', - return_value=[device_utils.DeviceUtils('123')]): + with mock.patch( + 'devil.android.device_utils.DeviceUtils.HealthyDevices', + return_value=[device_utils.DeviceUtils('123')]): with self.assertRaises(device_errors.DeviceUnreachableError): script_common.GetDevices(['456'], None) def testNoDevices(self): - with mock.patch('devil.android.device_utils.DeviceUtils.HealthyDevices', - return_value=[]): + with mock.patch( + 'devil.android.device_utils.DeviceUtils.HealthyDevices', + return_value=[]): with self.assertRaises(device_errors.NoDevicesError): script_common.GetDevices(None, None) class InitializeEnvironmentTest(unittest.TestCase): - def setUp(self): # pylint: disable=protected-access self.parser = argparse.ArgumentParser() @@ -77,9 +75,7 @@ class InitializeEnvironmentTest(unittest.TestCase): with tempfile.NamedTemporaryFile() as f: args = self.parser.parse_args(['--adb-path=%s' % f.name]) script_common.InitializeEnvironment(args) - self.assertEquals( - f.name, - devil_env.config.LocalPath('adb')) + self.assertEquals(f.name, devil_env.config.LocalPath('adb')) def testNonExistentAdb(self): with tempfile.NamedTemporaryFile() as f: diff --git a/catapult/devil/devil/android/tools/system_app.py b/catapult/devil/devil/android/tools/system_app.py index 8629ae68..c2f058e0 100755 --- a/catapult/devil/devil/android/tools/system_app.py +++ b/catapult/devil/devil/android/tools/system_app.py @@ -2,7 +2,6 @@ # Copyright 2017 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A script to replace a system app while running a command.""" import argparse @@ -13,14 +12,13 @@ import posixpath import re import sys - if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) - + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil.android import apk_helper +from devil.android import decorators from devil.android import device_errors from devil.android import device_temp_file from devil.android.sdk import version_codes @@ -31,20 +29,25 @@ from devil.utils import run_tests_helper logger = logging.getLogger(__name__) - # Some system apps aren't actually installed in the /system/ directory, so # special case them here with the correct install location. SPECIAL_SYSTEM_APP_LOCATIONS = { - # This also gets installed in /data/app when not a system app, so this script - # will remove either version. This doesn't appear to cause any issues, but - # will cause a few unnecessary reboots if this is the only package getting - # removed and it's already not a system app. - 'com.google.ar.core': '/data/app/', + # Older versions of ArCore were installed in /data/app/ regardless of + # whether they were system apps or not. Newer versions install in /system/ + # if they are system apps, and in /data/app/ if they aren't. + 'com.google.ar.core': ['/data/app/', '/system/'], + # On older versions of VrCore, the system app version is installed in + # /system/ like normal. However, at some point, this moved to /data/. + # So, we have to handle both cases. Like ArCore, this means we'll end up + # removing even non-system versions due to this, but it doesn't cause any + # issues. + 'com.google.vr.core': ['/data/app/', '/system/'], } # Gets app path and package name pm list packages -f output. _PM_LIST_PACKAGE_PATH_RE = re.compile(r'^\s*package:(\S+)=(\S+)\s*$') + def RemoveSystemApps(device, package_names): """Removes the given system apps. @@ -60,7 +63,9 @@ def RemoveSystemApps(device, package_names): @contextlib.contextmanager -def ReplaceSystemApp(device, package_name, replacement_apk, +def ReplaceSystemApp(device, + package_name, + replacement_apk, install_timeout=None): """A context manager that replaces the given system app while in scope. @@ -94,8 +99,8 @@ def _FindSystemPackagePaths(device, system_package_list): # TODO(aluo): Move this into device_utils.py def _GetApplicationPaths(device, package): paths = [] - lines = device.RunShellCommand(['pm', 'list', 'packages', '-f', '-u', - package], check_return=True) + lines = device.RunShellCommand( + ['pm', 'list', 'packages', '-f', '-u', package], check_return=True) for line in lines: match = re.match(_PM_LIST_PACKAGE_PATH_RE, line) if match: @@ -108,24 +113,34 @@ def _GetApplicationPaths(device, package): def _GetSystemPath(package, paths): for p in paths: - if p.startswith(SPECIAL_SYSTEM_APP_LOCATIONS.get(package, '/system/')): - return p + app_locations = SPECIAL_SYSTEM_APP_LOCATIONS.get(package, + ['/system/', '/product/']) + for location in app_locations: + if p.startswith(location): + return p return None +_MODIFICATION_TIMEOUT = 300 +_MODIFICATION_RETRIES = 2 _ENABLE_MODIFICATION_PROP = 'devil.modify_sys_apps' -@contextlib.contextmanager -def EnableSystemAppModification(device): - """A context manager that allows system apps to be modified while in scope. +def _ShouldRetryModification(exc): + return not isinstance(exc, device_errors.CommandTimeoutError) - Args: - device: (device_utils.DeviceUtils) the device - """ - if device.GetProp(_ENABLE_MODIFICATION_PROP) == '1': - yield - return + +# timeout and retries are both required by the decorator, but neither +# are used within the body of the function. +# pylint: disable=unused-argument + + +@decorators.WithTimeoutAndConditionalRetries(_ShouldRetryModification) +def _SetUpSystemAppModification(device, timeout=None, retries=None): + # Ensure that the device is online & available before proceeding to + # handle the case where something fails in the middle of set up and + # triggers a retry. + device.WaitUntilFullyBooted() # All calls that could potentially need root should run with as_root=True, but # it looks like some parts of Telemetry work as-is by implicitly assuming that @@ -151,13 +166,61 @@ def EnableSystemAppModification(device): device.adb.Remount() device.RunShellCommand(['stop'], check_return=True) device.SetProp(_ENABLE_MODIFICATION_PROP, '1') - yield - finally: + except device_errors.CommandFailedError: + if device.adb.is_emulator: + # Point the user to documentation, since there's a good chance they can + # workaround this on an emulator. + docs_url = ('https://chromium.googlesource.com/chromium/src/+/' + 'master/docs/android_emulator.md#writable-system-partition') + logger.error( + 'Did you start the emulator with "-writable-system?"\n' + 'See %s\n', docs_url) + raise + + return should_restore_root + + +@decorators.WithTimeoutAndConditionalRetries(_ShouldRetryModification) +def _TearDownSystemAppModification(device, + should_restore_root, + timeout=None, + retries=None): + try: device.SetProp(_ENABLE_MODIFICATION_PROP, '0') device.Reboot() device.WaitUntilFullyBooted() if should_restore_root: device.EnableRoot() + except device_errors.CommandTimeoutError: + logger.error('Timed out while tearing down system app modification.') + logger.error(' device state: %s', device.adb.GetState()) + raise + + +# pylint: enable=unused-argument + + +@contextlib.contextmanager +def EnableSystemAppModification(device): + """A context manager that allows system apps to be modified while in scope. + + Args: + device: (device_utils.DeviceUtils) the device + """ + if device.GetProp(_ENABLE_MODIFICATION_PROP) == '1': + yield + return + + should_restore_root = _SetUpSystemAppModification( + device, timeout=_MODIFICATION_TIMEOUT, retries=_MODIFICATION_RETRIES) + try: + yield + finally: + _TearDownSystemAppModification( + device, + should_restore_root, + timeout=_MODIFICATION_TIMEOUT, + retries=_MODIFICATION_RETRIES) @contextlib.contextmanager @@ -171,11 +234,9 @@ def _RelocateApp(device, package_name, relocate_to): for p in system_package_paths } relocation_dirs = [ - posixpath.dirname(d) - for _, d in relocation_map.iteritems() + posixpath.dirname(d) for _, d in relocation_map.iteritems() ] - device.RunShellCommand(['mkdir', '-p'] + relocation_dirs, - check_return=True) + device.RunShellCommand(['mkdir', '-p'] + relocation_dirs, check_return=True) _MoveApp(device, relocation_map) else: logger.info('No system package "%s"', package_name) @@ -207,10 +268,7 @@ def _MoveApp(device, relocation_map): device: (device_utils.DeviceUtils) relocation_map: (dict) A dict that maps src to dest """ - movements = [ - 'mv %s %s' % (k, v) - for k, v in relocation_map.iteritems() - ] + movements = ['mv %s %s' % (k, v) for k, v in relocation_map.iteritems()] cmd = ' && '.join(movements) with EnableSystemAppModification(device): device.RunShellCommand(cmd, as_root=True, check_return=True, shell=True) @@ -224,7 +282,10 @@ def main(raw_args): script_common.AddDeviceArguments(p) script_common.AddEnvironmentArguments(p) p.add_argument( - '-v', '--verbose', action='count', default=0, + '-v', + '--verbose', + action='count', + default=0, help='Print more information.') p.add_argument('command', nargs='*') @@ -235,7 +296,10 @@ def main(raw_args): remove_parser = subparsers.add_parser('remove') remove_parser.add_argument( - '--package', dest='packages', nargs='*', required=True, + '--package', + dest='packages', + nargs='*', + required=True, help='The system package(s) to remove.') add_common_arguments(remove_parser) remove_parser.set_defaults(func=remove_system_app) @@ -247,10 +311,11 @@ def main(raw_args): replace_parser = subparsers.add_parser('replace') replace_parser.add_argument( - '--package', required=True, - help='The system package to replace.') + '--package', required=True, help='The system package to replace.') replace_parser.add_argument( - '--replace-with', metavar='APK', required=True, + '--replace-with', + metavar='APK', + required=True, help='The APK with which the existing system app should be replaced.') add_common_arguments(replace_parser) replace_parser.set_defaults(func=replace_system_app) @@ -260,7 +325,7 @@ def main(raw_args): run_tests_helper.SetLogLevel(args.verbose) script_common.InitializeEnvironment(args) - devices = script_common.GetDevices(args.devices, args.blacklist_file) + devices = script_common.GetDevices(args.devices, args.denylist_file) parallel_devices = parallelizer.SyncParallelizer( [args.func(d, args) for d in devices]) with parallel_devices: diff --git a/catapult/devil/devil/android/tools/system_app_devicetest.py b/catapult/devil/devil/android/tools/system_app_devicetest.py index 293bad19..65faeea7 100755 --- a/catapult/devil/devil/android/tools/system_app_devicetest.py +++ b/catapult/devil/devil/android/tools/system_app_devicetest.py @@ -13,8 +13,8 @@ import unittest if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join( - os.path.dirname(__file__), '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil import base_error from devil import devil_env @@ -55,9 +55,9 @@ class SystemAppDeviceTest(device_test_case.DeviceTestCase): try: with device_temp_file.DeviceTempFile(self._device.adb) as tmp: self._device.adb.Push(cached_apk, tmp.name) - self._device.RunShellCommand( - ['mv', tmp.name, install_path], - as_root=True, check_return=True) + self._device.RunShellCommand(['mv', tmp.name, install_path], + as_root=True, + check_return=True) except base_error.BaseError: logger.warning('Failed to reinstall %s', os.path.basename(cached_apk)) @@ -71,11 +71,11 @@ class SystemAppDeviceTest(device_test_case.DeviceTestCase): def _check_preconditions(self): if not self._original_paths: - self.skipTest('%s is not installed on %s' % ( - self.PACKAGE, str(self._device))) + self.skipTest( + '%s is not installed on %s' % (self.PACKAGE, str(self._device))) if not any(p.startswith('/system/') for p in self._original_paths): - self.skipTest('%s is not installed in a system location on %s' % ( - self.PACKAGE, str(self._device))) + self.skipTest('%s is not installed in a system location on %s' % + (self.PACKAGE, str(self._device))) def testReplace(self): self._check_preconditions() diff --git a/catapult/devil/devil/android/tools/system_app_test.py b/catapult/devil/devil/android/tools/system_app_test.py index 44df7ead..1f1cb36e 100644 --- a/catapult/devil/devil/android/tools/system_app_test.py +++ b/catapult/devil/devil/android/tools/system_app_test.py @@ -8,8 +8,9 @@ import sys import unittest if __name__ == '__main__': - sys.path.append(os.path.abspath( - os.path.join(os.path.dirname(__file__), '..', '..', '..'))) + sys.path.append( + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil import devil_env from devil.android import device_utils @@ -20,19 +21,21 @@ from devil.android.tools import system_app with devil_env.SysPath(devil_env.PYMOCK_PATH): import mock - _PACKAGE_NAME = 'com.android' _PACKAGE_PATH = '/path/to/com.android.apk' -_PM_LIST_PACKAGES_COMMAND = ['pm', 'list', 'packages', '-f', '-u', - _PACKAGE_NAME] -_PM_LIST_PACKAGES_OUTPUT_WITH_PATH = ['package:/path/to/other=' + _PACKAGE_NAME - + '.other', 'package:' + _PACKAGE_PATH + - '=' + _PACKAGE_NAME] -_PM_LIST_PACKAGES_OUTPUT_WITHOUT_PATH = ['package:/path/to/other=' + - _PACKAGE_NAME + '.other'] +_PM_LIST_PACKAGES_COMMAND = [ + 'pm', 'list', 'packages', '-f', '-u', _PACKAGE_NAME +] +_PM_LIST_PACKAGES_OUTPUT_WITH_PATH = [ + 'package:/path/to/other=' + _PACKAGE_NAME + '.other', + 'package:' + _PACKAGE_PATH + '=' + _PACKAGE_NAME +] +_PM_LIST_PACKAGES_OUTPUT_WITHOUT_PATH = [ + 'package:/path/to/other=' + _PACKAGE_NAME + '.other' +] -class SystemAppTest(unittest.TestCase): +class SystemAppTest(unittest.TestCase): def testDoubleEnableModification(self): """Ensures that system app modification logic isn't repeated. @@ -79,8 +82,7 @@ class SystemAppTest(unittest.TestCase): # pylint: disable=protected-access mock_device = mock.Mock(spec=device_utils.DeviceUtils) mock_device.RunShellCommand.configure_mock( - return_value=_PM_LIST_PACKAGES_OUTPUT_WITH_PATH - ) + return_value=_PM_LIST_PACKAGES_OUTPUT_WITH_PATH) paths = system_app._GetApplicationPaths(mock_device, _PACKAGE_NAME) @@ -93,8 +95,7 @@ class SystemAppTest(unittest.TestCase): # pylint: disable=protected-access mock_device = mock.Mock(spec=device_utils.DeviceUtils) mock_device.RunShellCommand.configure_mock( - return_value=_PM_LIST_PACKAGES_OUTPUT_WITHOUT_PATH - ) + return_value=_PM_LIST_PACKAGES_OUTPUT_WITHOUT_PATH) paths = system_app._GetApplicationPaths(mock_device, _PACKAGE_NAME) @@ -106,9 +107,7 @@ class SystemAppTest(unittest.TestCase): """Nothing containing text of package name found in output.""" # pylint: disable=protected-access mock_device = mock.Mock(spec=device_utils.DeviceUtils) - mock_device.RunShellCommand.configure_mock( - return_value=[] - ) + mock_device.RunShellCommand.configure_mock(return_value=[]) paths = system_app._GetApplicationPaths(mock_device, _PACKAGE_NAME) @@ -121,8 +120,7 @@ class SystemAppTest(unittest.TestCase): # pylint: disable=protected-access mock_device = mock.Mock(spec=device_utils.DeviceUtils) mock_device.RunShellCommand.configure_mock( - return_value=_PM_LIST_PACKAGES_OUTPUT_WITH_PATH - ) + return_value=_PM_LIST_PACKAGES_OUTPUT_WITH_PATH) paths = system_app._GetApplicationPaths(mock_device, '') diff --git a/catapult/devil/devil/android/tools/unlock_bootloader.py b/catapult/devil/devil/android/tools/unlock_bootloader.py index b38f6690..96870581 100644 --- a/catapult/devil/devil/android/tools/unlock_bootloader.py +++ b/catapult/devil/devil/android/tools/unlock_bootloader.py @@ -2,7 +2,6 @@ # Copyright 2017 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A script to open the unlock bootloader on-screen prompt on all devices.""" import argparse @@ -14,8 +13,8 @@ import time if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil import devil_env from devil.android import device_errors @@ -71,11 +70,9 @@ def unlock_bootloader(d): cmd_old = [d._fastboot_path.read(), '-s', str(d), 'oem', 'unlock'] cmd_new = [d._fastboot_path.read(), '-s', str(d), 'flashing', 'unlock'] unlocking_processes.append( - subprocess.Popen( - cmd_old, stdout=subprocess.PIPE, stderr=subprocess.PIPE)) + subprocess.Popen(cmd_old, stdout=subprocess.PIPE, stderr=subprocess.PIPE)) unlocking_processes.append( - subprocess.Popen( - cmd_new, stdout=subprocess.PIPE, stderr=subprocess.PIPE)) + subprocess.Popen(cmd_new, stdout=subprocess.PIPE, stderr=subprocess.PIPE)) # Give the unlocking command time to finish and/or open the on-screen prompt. logging.info('Sleeping for 5 seconds...') @@ -95,8 +92,8 @@ def unlock_bootloader(d): logging.info('Device %s is waiting for confirmation.', d) else: logging.error( - 'Device %s is hanging, but not waiting for confirmation: %s', - d, out) + 'Device %s is hanging, but not waiting for confirmation: %s', d, + out) leftover_pids.append(p.pid) else: if 'unknown command' in out: @@ -123,21 +120,22 @@ def main(): parser = argparse.ArgumentParser() script_common.AddDeviceArguments(parser) - parser.add_argument('--adb-path', - help='Absolute path to the adb binary to use.') + parser.add_argument( + '--adb-path', help='Absolute path to the adb binary to use.') args = parser.parse_args() devil_dynamic_config = devil_env.EmptyConfig() if args.adb_path: devil_dynamic_config['dependencies'].update( - devil_env.LocalConfigItem( - 'adb', devil_env.GetPlatform(), args.adb_path)) + devil_env.LocalConfigItem('adb', devil_env.GetPlatform(), + args.adb_path)) devil_env.config.Initialize(configs=[devil_dynamic_config]) reboot_into_bootloader(args.devices) devices = [ - d for d in fastboot.Fastboot.Devices() if not args.devices or - str(d) in args.devices] + d for d in fastboot.Fastboot.Devices() + if not args.devices or str(d) in args.devices + ] parallel_devices = parallelizer.Parallelizer(devices) parallel_devices.pMap(unlock_bootloader).pGet(None) return 0 diff --git a/catapult/devil/devil/android/tools/video_recorder.py b/catapult/devil/devil/android/tools/video_recorder.py index 08432640..984931f3 100755 --- a/catapult/devil/devil/android/tools/video_recorder.py +++ b/catapult/devil/devil/android/tools/video_recorder.py @@ -2,7 +2,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Captures a video from an Android device.""" import argparse @@ -13,8 +12,9 @@ import time import sys if __name__ == '__main__': - sys.path.append(os.path.abspath(os.path.join( - os.path.dirname(__file__), '..', '..', '..'))) + sys.path.append( + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil.android import device_signal from devil.android import device_utils from devil.android.tools import script_common @@ -28,8 +28,7 @@ logger = logging.getLogger(__name__) class VideoRecorder(object): """Records a screen capture video from an Android Device (KitKat or newer).""" - def __init__(self, device, megabits_per_second=4, size=None, - rotate=False): + def __init__(self, device, megabits_per_second=4, size=None, rotate=False): """Creates a VideoRecorder instance. Args: @@ -44,7 +43,7 @@ class VideoRecorder(object): self._bit_rate = megabits_per_second * 1000 * 1000 self._device = device self._device_file = ( - '%s/screen-recording.mp4' % device.GetExternalStoragePath()) + '%s/screen-recording.mp4' % device.GetAppWritablePath()) self._recorder_thread = None self._rotate = rotate self._size = size @@ -55,6 +54,7 @@ class VideoRecorder(object): def Start(self, timeout=None): """Start recording video.""" + def screenrecord_started(): return bool(self._device.GetPids('screenrecord')) @@ -85,8 +85,8 @@ class VideoRecorder(object): def Stop(self): """Stop recording video.""" - if not self._device.KillAll('screenrecord', signum=device_signal.SIGINT, - quiet=True): + if not self._device.KillAll( + 'screenrecord', signum=device_signal.SIGINT, quiet=True): logger.warning('Nothing to kill: screenrecord was not running') self._recorder_thread.join() @@ -100,11 +100,8 @@ class VideoRecorder(object): """ # TODO(jbudorick): Merge filename generation with the logic for doing so in # DeviceUtils. - host_file_name = ( - host_file - or 'screen-recording-%s-%s.mp4' % ( - str(self._device), - time.strftime('%Y%m%dT%H%M%S', time.localtime()))) + host_file_name = (host_file or 'screen-recording-%s-%s.mp4' % (str( + self._device), time.strftime('%Y%m%dT%H%M%S', time.localtime()))) host_file_name = os.path.abspath(host_file_name) self._device.PullFile(self._device_file, host_file_name) self._device.RemovePath(self._device_file, force=True) @@ -114,24 +111,34 @@ class VideoRecorder(object): def main(): # Parse options. parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('-d', '--device', dest='devices', action='append', - help='Serial number of Android device to use.') - parser.add_argument('--blacklist-file', help='Device blacklist JSON file.') - parser.add_argument('-f', '--file', metavar='FILE', - help='Save result to file instead of generating a ' - 'timestamped file name.') - parser.add_argument('-v', '--verbose', action='store_true', - help='Verbose logging.') - parser.add_argument('-b', '--bitrate', default=4, type=float, - help='Bitrate in megabits/s, from 0.1 to 100 mbps, ' - '%(default)d mbps by default.') - parser.add_argument('-r', '--rotate', action='store_true', - help='Rotate video by 90 degrees.') - parser.add_argument('-s', '--size', metavar='WIDTHxHEIGHT', - help='Frame size to use instead of the device ' - 'screen size.') - parser.add_argument('host_file', nargs='?', - help='File to which the video capture will be written.') + script_common.AddDeviceArguments(parser) + parser.add_argument( + '-f', + '--file', + metavar='FILE', + help='Save result to file instead of generating a ' + 'timestamped file name.') + parser.add_argument( + '-v', '--verbose', action='store_true', help='Verbose logging.') + parser.add_argument( + '-b', + '--bitrate', + default=4, + type=float, + help='Bitrate in megabits/s, from 0.1 to 100 mbps, ' + '%(default)d mbps by default.') + parser.add_argument( + '-r', '--rotate', action='store_true', help='Rotate video by 90 degrees.') + parser.add_argument( + '-s', + '--size', + metavar='WIDTHxHEIGHT', + help='Frame size to use instead of the device ' + 'screen size.') + parser.add_argument( + 'host_file', + nargs='?', + help='File to which the video capture will be written.') args = parser.parse_args() @@ -140,9 +147,7 @@ def main(): if args.verbose: logging.getLogger().setLevel(logging.DEBUG) - size = (tuple(int(i) for i in args.size.split('x')) - if args.size - else None) + size = (tuple(int(i) for i in args.size.split('x')) if args.size else None) def record_video(device, stop_recording): recorder = VideoRecorder( @@ -157,9 +162,9 @@ def main(): f = recorder.Pull(f) print 'Video written to %s' % os.path.abspath(f) - parallel_devices = device_utils.DeviceUtils.parallel( - script_common.GetDevices(args.devices, args.blacklist_file), - async=True) + parallel_devices = device_utils.DeviceUtils.parallel(script_common.GetDevices( + args.devices, args.denylist_file), + async=True) stop_recording = threading.Event() running_recording = parallel_devices.pMap(record_video, stop_recording) print 'Recording. Press Enter to stop.', diff --git a/catapult/devil/devil/android/tools/wait_for_devices.py b/catapult/devil/devil/android/tools/wait_for_devices.py index bc733355..062f788f 100755 --- a/catapult/devil/devil/android/tools/wait_for_devices.py +++ b/catapult/devil/devil/android/tools/wait_for_devices.py @@ -2,7 +2,6 @@ # Copyright 2016 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Waits for the given devices to be available.""" import argparse @@ -11,8 +10,8 @@ import sys if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) from devil.android import device_utils from devil.android.tools import script_common @@ -22,11 +21,18 @@ from devil.utils import run_tests_helper def main(raw_args): parser = argparse.ArgumentParser() parser.add_argument('-v', '--verbose', action='count', help='Log more.') - parser.add_argument('-t', '--timeout', default=30, type=int, - help='Seconds to wait for the devices.') + parser.add_argument( + '-t', + '--timeout', + default=30, + type=int, + help='Seconds to wait for the devices.') parser.add_argument('--adb-path', help='ADB binary to use.') - parser.add_argument('device_serials', nargs='*', metavar='SERIAL', - help='Serials of the devices to wait for.') + parser.add_argument( + 'device_serials', + nargs='*', + metavar='SERIAL', + help='Serials of the devices to wait for.') args = parser.parse_args(raw_args) diff --git a/catapult/devil/devil/android/tools/webview_app.py b/catapult/devil/devil/android/tools/webview_app.py index 36b70391..406de1cb 100755 --- a/catapult/devil/devil/android/tools/webview_app.py +++ b/catapult/devil/devil/android/tools/webview_app.py @@ -2,7 +2,6 @@ # Copyright 2019 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A script to use a package as the WebView provider while running a command.""" import argparse @@ -12,14 +11,15 @@ import os import re import sys - if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..'))) - sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..', '..', '..', 'common', 'py_utils'))) - + os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..', '..'))) + sys.path.append( + os.path.abspath( + os.path.join( + os.path.dirname(__file__), '..', '..', '..', '..', 'common', + 'py_utils'))) from devil.android import apk_helper from devil.android import device_errors @@ -33,9 +33,10 @@ from py_utils import tempfile_ext logger = logging.getLogger(__name__) -_SYSTEM_PATH_RE = re.compile(r'^\s*\/system\/') +_SYSTEM_PATH_RE = re.compile(r'^\s*\/(system|product)\/') _WEBVIEW_INSTALL_TIMEOUT = 300 + @contextlib.contextmanager def UseWebViewProvider(device, apk, expected_package=''): """A context manager that uses the apk as the webview provider while in scope. @@ -55,8 +56,9 @@ def UseWebViewProvider(device, apk, expected_package=''): 'WebView Provider package %s does not match expected %s' % (package_name, expected_package), str(device)) - if (device.build_version_sdk in - [version_codes.NOUGAT, version_codes.NOUGAT_MR1]): + if (device.build_version_sdk in [ + version_codes.NOUGAT, version_codes.NOUGAT_MR1 + ]): logger.warning('Due to webviewupdate bug in Nougat, WebView Fallback Logic ' 'will be disabled and WebView provider may be changed after ' 'exit of UseWebViewProvider context manager scope.') @@ -84,9 +86,7 @@ def UseWebViewProvider(device, apk, expected_package=''): elif system_paths: # app is system app, use ReplaceSystemApp to install with system_app.ReplaceSystemApp( - device, - package_name, - apk, + device, package_name, apk, install_timeout=_WEBVIEW_INSTALL_TIMEOUT): _SetWebViewProvider(device, package_name) yield @@ -138,15 +138,15 @@ def _UninstallNonSystemApp(device, package_name): for user_path in user_paths: host_path = _RebasePath(temp_dir, user_path) # PullFile takes care of host_path creation if needed. - device.PullFile(user_path, host_path) + device.PullFile(user_path, host_path, timeout=_WEBVIEW_INSTALL_TIMEOUT) host_paths.append(host_path) device.Uninstall(package_name) try: yield finally: for host_path in reversed(host_paths): - device.Install(host_path, reinstall=True, - timeout=_WEBVIEW_INSTALL_TIMEOUT) + device.Install( + host_path, reinstall=True, timeout=_WEBVIEW_INSTALL_TIMEOUT) else: yield @@ -169,7 +169,10 @@ def main(raw_args): script_common.AddDeviceArguments(p) script_common.AddEnvironmentArguments(p) p.add_argument( - '-v', '--verbose', action='count', default=0, + '-v', + '--verbose', + action='count', + default=0, help='Print more information.') p.add_argument('command', nargs='*') @@ -179,10 +182,10 @@ def main(raw_args): yield parser.add_argument( - '--apk', required=True, - help='The apk to use as the provider.') + '--apk', required=True, help='The apk to use as the provider.') parser.add_argument( - '--expected-package', default='', + '--expected-package', + default='', help="Verify apk's package name matches value, disabled by default.") add_common_arguments(parser) parser.set_defaults(func=use_webview_provider) @@ -192,7 +195,7 @@ def main(raw_args): run_tests_helper.SetLogLevel(args.verbose) script_common.InitializeEnvironment(args) - devices = script_common.GetDevices(args.devices, args.blacklist_file) + devices = script_common.GetDevices(args.devices, args.denylist_file) parallel_devices = parallelizer.SyncParallelizer( [args.func(d, args) for d in devices]) with parallel_devices: diff --git a/catapult/devil/devil/android/valgrind_tools/base_tool.py b/catapult/devil/devil/android/valgrind_tools/base_tool.py index 2e6e9af3..b8445fcc 100644 --- a/catapult/devil/devil/android/valgrind_tools/base_tool.py +++ b/catapult/devil/devil/android/valgrind_tools/base_tool.py @@ -5,6 +5,7 @@ class BaseTool(object): """A tool that does nothing.""" + # pylint: disable=R0201 def __init__(self): diff --git a/catapult/devil/devil/base_error.py b/catapult/devil/devil/base_error.py index 4b896613..b7b33ccf 100644 --- a/catapult/devil/devil/base_error.py +++ b/catapult/devil/devil/base_error.py @@ -21,4 +21,3 @@ class BaseError(Exception): def is_infra_error(self): """Property to indicate if error was caused by an infrastructure issue.""" return self._is_infra_error - diff --git a/catapult/devil/devil/constants/exit_codes.py b/catapult/devil/devil/constants/exit_codes.py index aaeca4a8..a2deac69 100644 --- a/catapult/devil/devil/constants/exit_codes.py +++ b/catapult/devil/devil/constants/exit_codes.py @@ -1,7 +1,6 @@ # Copyright (c) 2012 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Common exit codes used by devil.""" ERROR = 1 diff --git a/catapult/devil/devil/devil_dependencies.json b/catapult/devil/devil/devil_dependencies.json index 8b397882..8e15e357 100644 --- a/catapult/devil/devil/devil_dependencies.json +++ b/catapult/devil/devil/devil_dependencies.json @@ -31,12 +31,22 @@ } } }, + "bundletool": { + "cloud_storage_base_folder": "binary_dependencies", + "cloud_storage_bucket": "chromium-telemetry", + "file_info": { + "default": { + "cloud_storage_hash": "721525531f3c3ec66b39afd74c2a05e6d3973eef", + "download_path": "../bin/deps/default/lib.java/bundletool-all-0.10.3.jar" + } + } + }, "chromium_commands": { "cloud_storage_base_folder": "binary_dependencies", "cloud_storage_bucket": "chromium-telemetry", "file_info": { "linux2_x86_64": { - "cloud_storage_hash": "4e22f641e4757309510e8d9f933f5aa504574ab6", + "cloud_storage_hash": "6efb1501c7c49e6ae84cc467648930eaf9f6a9e5", "download_path": "../bin/deps/linux2/x86_64/lib.java/chromium_commands.dex.jar" } } @@ -70,7 +80,7 @@ "cloud_storage_bucket": "chromium-telemetry", "file_info": { "linux2_x86_64": { - "cloud_storage_hash": "db9728166f182800eb9d09e9f036d56e105e8235", + "cloud_storage_hash": "fe84a6714192f6596bf09215caeea34ed937a42b", "download_path": "../bin/deps/linux2/x86_64/bin/fastboot" } } @@ -80,12 +90,20 @@ "cloud_storage_bucket": "chromium-telemetry", "file_info": { "android_arm64-v8a": { - "cloud_storage_hash": "f222268d8442979240d1b18de00911a49e548daa", + "cloud_storage_hash": "b30e8bfb234d02899b05a7957c6a84dc6e16ec4b", "download_path": "../bin/deps/android/arm64-v8a/bin/forwarder_device" }, "android_armeabi-v7a": { - "cloud_storage_hash": "c15267bf01c26eb0aea4f61c780bbba460c5c981", + "cloud_storage_hash": "efc7f092f5394a7b88d77a2544adfe33c5c4fd3c", "download_path": "../bin/deps/android/armeabi-v7a/bin/forwarder_device" + }, + "android_x86": { + "cloud_storage_hash": "ec841e99335f8c9bbcf2e3da40dd8a7ebefe2b6e", + "download_path": "../bin/deps/android/x86/bin/forwarder_device" + }, + "android_x86_64": { + "cloud_storage_hash": "b9921146385bb7c3fc7246084c37a175e7bb97bd", + "download_path": "../bin/deps/android/x86_64/bin/forwarder_device" } } }, @@ -94,7 +112,7 @@ "cloud_storage_bucket": "chromium-telemetry", "file_info": { "linux2_x86_64": { - "cloud_storage_hash": "8fe69994b670f028484eed475dbffc838c8a57f7", + "cloud_storage_hash": "334c172dd90bae568cbff9465130f1245a35cade", "download_path": "../bin/deps/linux2/x86_64/forwarder_host" } } @@ -104,16 +122,20 @@ "cloud_storage_bucket": "chromium-telemetry", "file_info": { "android_arm64-v8a": { - "cloud_storage_hash": "4e7d2dedd9c6321fdc152b06869e09a3c5817904", + "cloud_storage_hash": "e9e1a4e2f1c24180a5cac496c3c50213384b51eb", "download_path": "../bin/deps/android/arm64-v8a/bin/md5sum_device" }, "android_armeabi-v7a": { - "cloud_storage_hash": "39fd90af0f8828202b687f7128393759181c5e2e", + "cloud_storage_hash": "f0cf40aec71df24d50a2644bed199ffd7588082d", "download_path": "../bin/deps/android/armeabi-v7a/bin/md5sum_device" }, "android_x86": { - "cloud_storage_hash": "d5cf42ab5986a69c31c0177b0df499d6bf708df6", + "cloud_storage_hash": "7530812af4b3cd2f20d9ce27e3615828cb4e4fc2", "download_path": "../bin/deps/android/x86/bin/md5sum_device" + }, + "android_x86_64": { + "cloud_storage_hash": "4cd36953c44119578568a723854d43b8ed124d7e", + "download_path": "../bin/deps/android/x86_64/bin/md5sum_device" } } }, @@ -122,7 +144,7 @@ "cloud_storage_bucket": "chromium-telemetry", "file_info": { "linux2_x86_64": { - "cloud_storage_hash": "4db5bd5e9bea8880d8bf2caa59d0efb0acc19f74", + "cloud_storage_hash": "f2428a52730c6593267678c75f95cdc7dffb58df", "download_path": "../bin/deps/linux2/x86_64/bin/md5sum_host" } } diff --git a/catapult/devil/devil/devil_env.py b/catapult/devil/devil/devil_env.py index aa4fe1ee..b39e01c8 100644 --- a/catapult/devil/devil/devil_env.py +++ b/catapult/devil/devil/devil_env.py @@ -11,12 +11,10 @@ import sys import tempfile import threading -CATAPULT_ROOT_PATH = os.path.abspath(os.path.join( - os.path.dirname(__file__), '..', '..')) -DEPENDENCY_MANAGER_PATH = os.path.join( - CATAPULT_ROOT_PATH, 'dependency_manager') -PYMOCK_PATH = os.path.join( - CATAPULT_ROOT_PATH, 'third_party', 'mock') +CATAPULT_ROOT_PATH = os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..')) +DEPENDENCY_MANAGER_PATH = os.path.join(CATAPULT_ROOT_PATH, 'dependency_manager') +PYMOCK_PATH = os.path.join(CATAPULT_ROOT_PATH, 'third_party', 'mock') @contextlib.contextmanager @@ -28,52 +26,49 @@ def SysPath(path): else: sys.path.pop() + with SysPath(DEPENDENCY_MANAGER_PATH): import dependency_manager # pylint: disable=import-error _ANDROID_BUILD_TOOLS = {'aapt', 'dexdump', 'split-select'} -_DEVIL_DEFAULT_CONFIG = os.path.abspath(os.path.join( - os.path.dirname(__file__), 'devil_dependencies.json')) +_DEVIL_DEFAULT_CONFIG = os.path.abspath( + os.path.join(os.path.dirname(__file__), 'devil_dependencies.json')) _LEGACY_ENVIRONMENT_VARIABLES = { - 'ADB_PATH': { - 'dependency_name': 'adb', - 'platform': 'linux2_x86_64', - }, - 'ANDROID_SDK_ROOT': { - 'dependency_name': 'android_sdk', - 'platform': 'linux2_x86_64', - }, + 'ADB_PATH': { + 'dependency_name': 'adb', + 'platform': 'linux2_x86_64', + }, + 'ANDROID_SDK_ROOT': { + 'dependency_name': 'android_sdk', + 'platform': 'linux2_x86_64', + }, } def EmptyConfig(): - return { - 'config_type': 'BaseConfig', - 'dependencies': {} - } + return {'config_type': 'BaseConfig', 'dependencies': {}} def LocalConfigItem(dependency_name, dependency_platform, dependency_path): if isinstance(dependency_path, basestring): dependency_path = [dependency_path] return { - dependency_name: { - 'file_info': { - dependency_platform: { - 'local_paths': dependency_path - }, + dependency_name: { + 'file_info': { + dependency_platform: { + 'local_paths': dependency_path + }, + }, }, - }, } def _GetEnvironmentVariableConfig(): env_config = EmptyConfig() - path_config = ( - (os.environ.get(k), v) - for k, v in _LEGACY_ENVIRONMENT_VARIABLES.iteritems()) + path_config = ((os.environ.get(k), v) + for k, v in _LEGACY_ENVIRONMENT_VARIABLES.iteritems()) path_config = ((p, c) for p, c in path_config if p) for p, c in path_config: env_config['dependencies'].update( @@ -82,7 +77,6 @@ def _GetEnvironmentVariableConfig(): class _Environment(object): - def __init__(self): self._dm_init_lock = threading.Lock() self._dm = None @@ -111,9 +105,7 @@ class _Environment(object): env_config = _GetEnvironmentVariableConfig() if env_config: configs.insert(0, env_config) - self._InitializeRecursive( - configs=configs, - config_files=config_files) + self._InitializeRecursive(configs=configs, config_files=config_files) assert self._dm is not None, 'Failed to create dependency manager.' def _InitializeRecursive(self, configs=None, config_files=None): @@ -191,4 +183,3 @@ def GetPlatform(arch=None, device=None): config = _Environment() - diff --git a/catapult/devil/devil/devil_env_test.py b/catapult/devil/devil/devil_env_test.py index e78221a0..ee7cd8fd 100755 --- a/catapult/devil/devil/devil_env_test.py +++ b/catapult/devil/devil/devil_env_test.py @@ -19,12 +19,10 @@ _sys_path_after = list(sys.path) class DevilEnvTest(unittest.TestCase): - def testSysPath(self): self.assertEquals(_sys_path_before, _sys_path_after) - self.assertEquals( - _sys_path_before + [devil_env.PYMOCK_PATH], - _sys_path_with_pymock) + self.assertEquals(_sys_path_before + [devil_env.PYMOCK_PATH], + _sys_path_with_pymock) def testGetEnvironmentVariableConfig_configType(self): with mock.patch('os.environ.get', @@ -42,20 +40,17 @@ class DevilEnvTest(unittest.TestCase): def mock_environment(env_var): return '/my/fake/adb/path' if env_var == 'ADB_PATH' else None - with mock.patch('os.environ.get', - mock.Mock(side_effect=mock_environment)): + with mock.patch('os.environ.get', mock.Mock(side_effect=mock_environment)): env_config = devil_env._GetEnvironmentVariableConfig() - self.assertEquals( - { - 'adb': { + self.assertEquals({ + 'adb': { 'file_info': { - 'linux2_x86_64': { - 'local_paths': ['/my/fake/adb/path'], - }, + 'linux2_x86_64': { + 'local_paths': ['/my/fake/adb/path'], + }, }, - }, }, - env_config.get('dependencies')) + }, env_config.get('dependencies')) if __name__ == '__main__': diff --git a/catapult/devil/devil/utils/cmd_helper.py b/catapult/devil/devil/utils/cmd_helper.py index 3c4a06ed..634c9716 100644 --- a/catapult/devil/devil/utils/cmd_helper.py +++ b/catapult/devil/devil/utils/cmd_helper.py @@ -1,7 +1,6 @@ # Copyright (c) 2012 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A wrapper for subprocess to make calling shell commands easier.""" import codecs @@ -86,6 +85,7 @@ def ShrinkToSnippet(cmd_parts, var_name, var_value): Returns: A shell snippet that does not include setting the variable. """ + def shrink(value): parts = (x and SingleQuote(x) for x in value.split(var_value)) with_substitutions = ('"$%s"' % var_name).join(parts) @@ -94,23 +94,36 @@ def ShrinkToSnippet(cmd_parts, var_name, var_value): return ' '.join(shrink(part) for part in cmd_parts) -def Popen(args, stdout=None, stderr=None, shell=None, cwd=None, env=None): +def Popen(args, + stdin=None, + stdout=None, + stderr=None, + shell=None, + cwd=None, + env=None): # preexec_fn isn't supported on windows. if sys.platform == 'win32': - close_fds = (stdout is None and stderr is None) + close_fds = (stdin is None and stdout is None and stderr is None) preexec_fn = None else: close_fds = True preexec_fn = lambda: signal.signal(signal.SIGPIPE, signal.SIG_DFL) return subprocess.Popen( - args=args, cwd=cwd, stdout=stdout, stderr=stderr, - shell=shell, close_fds=close_fds, env=env, preexec_fn=preexec_fn) + args=args, + cwd=cwd, + stdin=stdin, + stdout=stdout, + stderr=stderr, + shell=shell, + close_fds=close_fds, + env=env, + preexec_fn=preexec_fn) def Call(args, stdout=None, stderr=None, shell=None, cwd=None, env=None): - pipe = Popen(args, stdout=stdout, stderr=stderr, shell=shell, cwd=cwd, - env=env) + pipe = Popen( + args, stdout=stdout, stderr=stderr, shell=shell, cwd=cwd, env=env) pipe.communicate() return pipe.wait() @@ -127,7 +140,7 @@ def RunCmd(args, cwd=None): Returns: Return code from the command execution. """ - logger.info(str(args) + ' ' + (cwd or '')) + logger.debug(str(args) + ' ' + (cwd or '')) return Call(args, cwd=cwd) @@ -167,7 +180,11 @@ def _ValidateAndLogCommand(args, cwd, shell): return args -def GetCmdStatusAndOutput(args, cwd=None, shell=False, env=None): +def GetCmdStatusAndOutput(args, + cwd=None, + shell=False, + env=None, + merge_stderr=False): """Executes a subprocess and returns its exit code and output. Args: @@ -179,12 +196,13 @@ def GetCmdStatusAndOutput(args, cwd=None, shell=False, env=None): is a string and False if args is a sequence. env: If not None, a mapping that defines environment variables for the subprocess. + merge_stderr: If True, captures stderr as part of stdout. Returns: The 2-tuple (exit code, stdout). """ status, stdout, stderr = GetCmdStatusOutputAndError( - args, cwd=cwd, shell=shell, env=env) + args, cwd=cwd, shell=shell, env=env, merge_stderr=merge_stderr) if stderr: logger.critical('STDERR: %s', stderr) @@ -210,11 +228,20 @@ def StartCmd(args, cwd=None, shell=False, env=None): A process handle from subprocess.Popen. """ _ValidateAndLogCommand(args, cwd, shell) - return Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, - shell=shell, cwd=cwd, env=env) - - -def GetCmdStatusOutputAndError(args, cwd=None, shell=False, env=None): + return Popen( + args, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + shell=shell, + cwd=cwd, + env=env) + + +def GetCmdStatusOutputAndError(args, + cwd=None, + shell=False, + env=None, + merge_stderr=False): """Executes a subprocess and returns its exit code, output, and errors. Args: @@ -226,13 +253,20 @@ def GetCmdStatusOutputAndError(args, cwd=None, shell=False, env=None): is a string and False if args is a sequence. env: If not None, a mapping that defines environment variables for the subprocess. + merge_stderr: If True, captures stderr as part of stdout. Returns: The 3-tuple (exit code, stdout, stderr). """ _ValidateAndLogCommand(args, cwd, shell) - pipe = Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, - shell=shell, cwd=cwd, env=env) + stderr = subprocess.STDOUT if merge_stderr else subprocess.PIPE + pipe = Popen( + args, + stdout=subprocess.PIPE, + stderr=stderr, + shell=shell, + cwd=cwd, + env=env) stdout, stderr = pipe.communicate() return (pipe.returncode, stdout, stderr) @@ -249,9 +283,11 @@ class TimeoutError(base_error.BaseError): return self._output -def _IterProcessStdoutFcntl( - process, iter_timeout=None, timeout=None, buffer_size=4096, - poll_interval=1): +def _IterProcessStdoutFcntl(process, + iter_timeout=None, + timeout=None, + buffer_size=4096, + poll_interval=1): """An fcntl-based implementation of _IterProcessStdout.""" # pylint: disable=too-many-nested-blocks import fcntl @@ -272,14 +308,13 @@ def _IterProcessStdoutFcntl( iter_end_time = time.time() + iter_timeout if iter_end_time: - iter_aware_poll_interval = min( - poll_interval, - max(0, iter_end_time - time.time())) + iter_aware_poll_interval = min(poll_interval, + max(0, iter_end_time - time.time())) else: iter_aware_poll_interval = poll_interval - read_fds, _, _ = select.select( - [child_fd], [], [], iter_aware_poll_interval) + read_fds, _, _ = select.select([child_fd], [], [], + iter_aware_poll_interval) if child_fd in read_fds: data = os.read(child_fd, buffer_size) if not data: @@ -290,8 +325,8 @@ def _IterProcessStdoutFcntl( # If process is closed, keep checking for output data (because of timing # issues). while True: - read_fds, _, _ = select.select( - [child_fd], [], [], iter_aware_poll_interval) + read_fds, _, _ = select.select([child_fd], [], [], + iter_aware_poll_interval) if child_fd in read_fds: data = os.read(child_fd, buffer_size) if data: @@ -310,9 +345,11 @@ def _IterProcessStdoutFcntl( process.wait() -def _IterProcessStdoutQueue( - process, iter_timeout=None, timeout=None, buffer_size=4096, - poll_interval=1): +def _IterProcessStdoutQueue(process, + iter_timeout=None, + timeout=None, + buffer_size=4096, + poll_interval=1): """A Queue.Queue-based implementation of _IterProcessStdout. TODO(jbudorick): Evaluate whether this is a suitable replacement for @@ -363,10 +400,8 @@ def _IterProcessStdoutQueue( reader_thread.join() -_IterProcessStdout = ( - _IterProcessStdoutQueue - if sys.platform == 'win32' - else _IterProcessStdoutFcntl) +_IterProcessStdout = (_IterProcessStdoutQueue + if sys.platform == 'win32' else _IterProcessStdoutFcntl) """Iterate over a process's stdout. This is intentionally not public. @@ -390,8 +425,12 @@ Yields: """ -def GetCmdStatusAndOutputWithTimeout(args, timeout, cwd=None, shell=False, - logfile=None, env=None): +def GetCmdStatusAndOutputWithTimeout(args, + timeout, + cwd=None, + shell=False, + logfile=None, + env=None): """Executes a subprocess with a timeout. Args: @@ -414,8 +453,13 @@ def GetCmdStatusAndOutputWithTimeout(args, timeout, cwd=None, shell=False, """ _ValidateAndLogCommand(args, cwd, shell) output = StringIO.StringIO() - process = Popen(args, cwd=cwd, shell=shell, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, env=env) + process = Popen( + args, + cwd=cwd, + shell=shell, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + env=env) try: for data in _IterProcessStdout(process, timeout=timeout): if logfile: @@ -430,8 +474,13 @@ def GetCmdStatusAndOutputWithTimeout(args, timeout, cwd=None, shell=False, return process.returncode, str_output -def IterCmdOutputLines(args, iter_timeout=None, timeout=None, cwd=None, - shell=False, env=None, check_status=True): +def IterCmdOutputLines(args, + iter_timeout=None, + timeout=None, + cwd=None, + shell=False, + env=None, + check_status=True): """Executes a subprocess and continuously yields lines from its output. Args: @@ -455,13 +504,25 @@ def IterCmdOutputLines(args, iter_timeout=None, timeout=None, cwd=None, non-zero exit status. """ cmd = _ValidateAndLogCommand(args, cwd, shell) - process = Popen(args, cwd=cwd, shell=shell, env=env, - stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + process = Popen( + args, + cwd=cwd, + shell=shell, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) return _IterCmdOutputLines( - process, cmd, iter_timeout=iter_timeout, timeout=timeout, + process, + cmd, + iter_timeout=iter_timeout, + timeout=timeout, check_status=check_status) -def _IterCmdOutputLines(process, cmd, iter_timeout=None, timeout=None, + +def _IterCmdOutputLines(process, + cmd, + iter_timeout=None, + timeout=None, check_status=True): buffer_output = '' @@ -471,8 +532,8 @@ def _IterCmdOutputLines(process, cmd, iter_timeout=None, timeout=None, iter_end = time.time() + iter_timeout cur_iter_timeout = iter_timeout - for data in _IterProcessStdout(process, iter_timeout=cur_iter_timeout, - timeout=timeout): + for data in _IterProcessStdout( + process, iter_timeout=cur_iter_timeout, timeout=timeout): if iter_timeout: # Check whether the current iteration has timed out. cur_iter_timeout = iter_end - time.time() diff --git a/catapult/devil/devil/utils/cmd_helper_test.py b/catapult/devil/devil/utils/cmd_helper_test.py index 6a8e8813..57abceb4 100755 --- a/catapult/devil/devil/utils/cmd_helper_test.py +++ b/catapult/devil/devil/utils/cmd_helper_test.py @@ -2,7 +2,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Tests for the cmd_helper module.""" import unittest @@ -18,14 +17,11 @@ with devil_env.SysPath(devil_env.PYMOCK_PATH): class CmdHelperSingleQuoteTest(unittest.TestCase): - def testSingleQuote_basic(self): - self.assertEquals('hello', - cmd_helper.SingleQuote('hello')) + self.assertEquals('hello', cmd_helper.SingleQuote('hello')) def testSingleQuote_withSpaces(self): - self.assertEquals("'hello world'", - cmd_helper.SingleQuote('hello world')) + self.assertEquals("'hello world'", cmd_helper.SingleQuote('hello world')) def testSingleQuote_withUnsafeChars(self): self.assertEquals("""'hello'"'"'; rm -rf /'""", @@ -39,14 +35,11 @@ class CmdHelperSingleQuoteTest(unittest.TestCase): class CmdHelperDoubleQuoteTest(unittest.TestCase): - def testDoubleQuote_basic(self): - self.assertEquals('hello', - cmd_helper.DoubleQuote('hello')) + self.assertEquals('hello', cmd_helper.DoubleQuote('hello')) def testDoubleQuote_withSpaces(self): - self.assertEquals('"hello world"', - cmd_helper.DoubleQuote('hello world')) + self.assertEquals('"hello world"', cmd_helper.DoubleQuote('hello world')) def testDoubleQuote_withUnsafeChars(self): self.assertEquals('''"hello\\"; rm -rf /"''', @@ -60,33 +53,34 @@ class CmdHelperDoubleQuoteTest(unittest.TestCase): class CmdHelperShinkToSnippetTest(unittest.TestCase): - def testShrinkToSnippet_noArgs(self): - self.assertEquals('foo', - cmd_helper.ShrinkToSnippet(['foo'], 'a', 'bar')) + self.assertEquals('foo', cmd_helper.ShrinkToSnippet(['foo'], 'a', 'bar')) self.assertEquals("'foo foo'", - cmd_helper.ShrinkToSnippet(['foo foo'], 'a', 'bar')) + cmd_helper.ShrinkToSnippet(['foo foo'], 'a', 'bar')) self.assertEquals('"$a"\' bar\'', - cmd_helper.ShrinkToSnippet(['foo bar'], 'a', 'foo')) + cmd_helper.ShrinkToSnippet(['foo bar'], 'a', 'foo')) self.assertEquals('\'foo \'"$a"', - cmd_helper.ShrinkToSnippet(['foo bar'], 'a', 'bar')) + cmd_helper.ShrinkToSnippet(['foo bar'], 'a', 'bar')) self.assertEquals('foo"$a"', - cmd_helper.ShrinkToSnippet(['foobar'], 'a', 'bar')) + cmd_helper.ShrinkToSnippet(['foobar'], 'a', 'bar')) def testShrinkToSnippet_singleArg(self): self.assertEquals("foo ''", - cmd_helper.ShrinkToSnippet(['foo', ''], 'a', 'bar')) + cmd_helper.ShrinkToSnippet(['foo', ''], 'a', 'bar')) self.assertEquals("foo foo", - cmd_helper.ShrinkToSnippet(['foo', 'foo'], 'a', 'bar')) + cmd_helper.ShrinkToSnippet(['foo', 'foo'], 'a', 'bar')) self.assertEquals('"$a" "$a"', - cmd_helper.ShrinkToSnippet(['foo', 'foo'], 'a', 'foo')) + cmd_helper.ShrinkToSnippet(['foo', 'foo'], 'a', 'foo')) self.assertEquals('foo "$a""$a"', - cmd_helper.ShrinkToSnippet(['foo', 'barbar'], 'a', 'bar')) - self.assertEquals('foo "$a"\' \'"$a"', + cmd_helper.ShrinkToSnippet(['foo', 'barbar'], 'a', 'bar')) + self.assertEquals( + 'foo "$a"\' \'"$a"', cmd_helper.ShrinkToSnippet(['foo', 'bar bar'], 'a', 'bar')) - self.assertEquals('foo "$a""$a"\' \'', + self.assertEquals( + 'foo "$a""$a"\' \'', cmd_helper.ShrinkToSnippet(['foo', 'barbar '], 'a', 'bar')) - self.assertEquals('foo \' \'"$a""$a"\' \'', + self.assertEquals( + 'foo \' \'"$a""$a"\' \'', cmd_helper.ShrinkToSnippet(['foo', ' barbar '], 'a', 'bar')) @@ -94,7 +88,6 @@ _DEFAULT = 'DEFAULT' class _ProcessOutputEvent(object): - def __init__(self, select_fds=_DEFAULT, read_contents=None, ts=_DEFAULT): self.select_fds = select_fds self.read_contents = read_contents @@ -102,7 +95,6 @@ class _ProcessOutputEvent(object): class _MockProcess(object): - def __init__(self, output_sequence=None, return_value=0): # Arbitrary. @@ -130,8 +122,7 @@ class _MockProcess(object): # Use an leading element to make the iteration logic work. initial_seq_element = _ProcessOutputEvent( - _DEFAULT, '', - output_sequence[0].ts if output_sequence else _DEFAULT) + _DEFAULT, '', output_sequence[0].ts if output_sequence else _DEFAULT) output_sequence.insert(0, initial_seq_element) for o in output_sequence: @@ -159,10 +150,10 @@ class _MockProcess(object): else: self._output_seq_index += 1 if self._output_seq_index < len(self._output_sequence): - return (self._output_sequence[self._output_seq_index].select_fds, - None, None) + return (self._output_sequence[self._output_seq_index].select_fds, None, + None) else: - return([], None, None) + return ([], None, None) def time_side_effect(*_args, **_kwargs): return self._output_sequence[self._output_seq_index].ts @@ -179,9 +170,9 @@ class _MockProcess(object): # Set up but *do not start* the mocks. self._mocks = [ - mock.patch('os.read', new=mock_read), - mock.patch('select.select', new=mock_select), - mock.patch('time.time', new=mock_time), + mock.patch('os.read', new=mock_read), + mock.patch('select.select', new=mock_select), + mock.patch('time.time', new=mock_time), ] if sys.platform != 'win32': self._mocks.append(mock.patch('fcntl.fcntl')) @@ -204,7 +195,7 @@ class CmdHelperIterCmdOutputLinesTest(unittest.TestCase): # pylint: disable=protected-access _SIMPLE_OUTPUT_SEQUENCE = [ - _ProcessOutputEvent(read_contents='1\n2\n'), + _ProcessOutputEvent(read_contents='1\n2\n'), ] def testIterCmdOutputLines_success(self): @@ -216,25 +207,27 @@ class CmdHelperIterCmdOutputLinesTest(unittest.TestCase): def testIterCmdOutputLines_exitStatusFail(self): with self.assertRaises(subprocess.CalledProcessError): - with _MockProcess(output_sequence=self._SIMPLE_OUTPUT_SEQUENCE, - return_value=1) as mock_proc: + with _MockProcess( + output_sequence=self._SIMPLE_OUTPUT_SEQUENCE, + return_value=1) as mock_proc: for num, line in enumerate( cmd_helper._IterCmdOutputLines(mock_proc, 'mock_proc'), 1): self.assertEquals(num, int(line)) # after reading all the output we get an exit status of 1 def testIterCmdOutputLines_exitStatusIgnored(self): - with _MockProcess(output_sequence=self._SIMPLE_OUTPUT_SEQUENCE, - return_value=1) as mock_proc: + with _MockProcess( + output_sequence=self._SIMPLE_OUTPUT_SEQUENCE, + return_value=1) as mock_proc: for num, line in enumerate( cmd_helper._IterCmdOutputLines( - mock_proc, 'mock_proc', check_status=False), - 1): + mock_proc, 'mock_proc', check_status=False), 1): self.assertEquals(num, int(line)) def testIterCmdOutputLines_exitStatusSkipped(self): - with _MockProcess(output_sequence=self._SIMPLE_OUTPUT_SEQUENCE, - return_value=1) as mock_proc: + with _MockProcess( + output_sequence=self._SIMPLE_OUTPUT_SEQUENCE, + return_value=1) as mock_proc: for num, line in enumerate( cmd_helper._IterCmdOutputLines(mock_proc, 'mock_proc'), 1): self.assertEquals(num, int(line)) @@ -245,14 +238,14 @@ class CmdHelperIterCmdOutputLinesTest(unittest.TestCase): def testIterCmdOutputLines_delay(self): output_sequence = [ - _ProcessOutputEvent(read_contents='1\n2\n', ts=1), - _ProcessOutputEvent(read_contents=None, ts=2), - _ProcessOutputEvent(read_contents='Awake', ts=10), + _ProcessOutputEvent(read_contents='1\n2\n', ts=1), + _ProcessOutputEvent(read_contents=None, ts=2), + _ProcessOutputEvent(read_contents='Awake', ts=10), ] with _MockProcess(output_sequence=output_sequence) as mock_proc: for num, line in enumerate( - cmd_helper._IterCmdOutputLines(mock_proc, 'mock_proc', - iter_timeout=5), 1): + cmd_helper._IterCmdOutputLines( + mock_proc, 'mock_proc', iter_timeout=5), 1): if num <= 2: self.assertEquals(num, int(line)) elif num == 3: diff --git a/catapult/devil/devil/utils/file_utils.py b/catapult/devil/devil/utils/file_utils.py index dc5a9efc..6bc9e95d 100644 --- a/catapult/devil/devil/utils/file_utils.py +++ b/catapult/devil/devil/utils/file_utils.py @@ -27,5 +27,3 @@ def MergeFiles(dest_file, source_files): except OSError: pass raise e - - diff --git a/catapult/devil/devil/utils/find_usb_devices.py b/catapult/devil/devil/utils/find_usb_devices.py index b6dcc702..eb45a6ce 100755 --- a/catapult/devil/devil/utils/find_usb_devices.py +++ b/catapult/devil/devil/utils/find_usb_devices.py @@ -11,8 +11,7 @@ import sys if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..'))) + os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))) from devil.utils import cmd_helper from devil.utils import usb_hubs @@ -267,13 +266,13 @@ def GetBusNumberToDeviceTreeMap(fast=True): for line in lsusb.raw_lsusb().splitlines(): match = _LSUSB_BUS_DEVICE_RE.match(line) if match: - info_map[(int(match.group(1)), int(match.group(2)))] = ( - {'desc':match.group(3)}) + info_map[(int(match.group(1)), int(match.group(2)))] = ({ + 'desc': match.group(3) + }) else: info_map = {((int(line['bus']), int(line['device']))): line for line in _GetParsedLSUSBOutput()} - tree = {} bus_num = -1 for line in _GetUSBDevicesOutput().splitlines(): @@ -290,10 +289,10 @@ def GetBusNumberToDeviceTreeMap(fast=True): tree[bus_num] = USBBusNode(bus_num=bus_num) # create the new device - new_device = USBDeviceNode(bus_num=bus_num, - device_num=device_num, - info=info_map.get((bus_num, device_num), - {'desc': 'NOT AVAILABLE'})) + new_device = USBDeviceNode( + bus_num=bus_num, + device_num=device_num, + info=info_map.get((bus_num, device_num), {'desc': 'NOT AVAILABLE'})) # add device to bus if parent_num != 0: @@ -351,8 +350,10 @@ def GetPhysicalPortToBusDeviceMap(hub, hub_type): Dict of {physical port: (bus number, device number)} """ port_device = hub_type.GetPhysicalPortToNodeTuples(hub) - return {port: (device.bus_num, device.device_num) - for (port, device) in port_device} + return { + port: (device.bus_num, device.device_num) + for (port, device) in port_device + } def GetPhysicalPortToSerialMap(hub, hub_type): @@ -366,9 +367,10 @@ def GetPhysicalPortToSerialMap(hub, hub_type): Dict of {physical port: serial number)} """ port_device = hub_type.GetPhysicalPortToNodeTuples(hub) - return {port: device.serial - for (port, device) in port_device - if device.serial} + return { + port: device.serial + for (port, device) in port_device if device.serial + } def GetPhysicalPortToTTYMap(device, hub_type): @@ -382,9 +384,11 @@ def GetPhysicalPortToTTYMap(device, hub_type): """ port_device = hub_type.GetPhysicalPortToNodeTuples(device) bus_device_to_tty = GetBusDeviceToTTYMap() - return {port: bus_device_to_tty[(device.bus_num, device.device_num)] - for (port, device) in port_device - if (device.bus_num, device.device_num) in bus_device_to_tty} + return { + port: bus_device_to_tty[(device.bus_num, device.device_num)] + for (port, device) in port_device + if (device.bus_num, device.device_num) in bus_device_to_tty + } def CollectHubMaps(hub_types, map_func, device_tree_map=None, fast=False): @@ -538,9 +542,11 @@ def parse_options(argv): parser = argparse.ArgumentParser(usage=USAGE) return parser.parse_args(argv[1:]) + def main(): parse_options(sys.argv) TestUSBTopologyScript() + if __name__ == "__main__": sys.exit(main()) diff --git a/catapult/devil/devil/utils/find_usb_devices_test.py b/catapult/devil/devil/utils/find_usb_devices_test.py index c22f21b8..a8620b62 100755 --- a/catapult/devil/devil/utils/find_usb_devices_test.py +++ b/catapult/devil/devil/utils/find_usb_devices_test.py @@ -4,7 +4,6 @@ # found in the LICENSE file. # pylint: disable=protected-access - """ Unit tests for the contents of find_usb_devices.py. @@ -37,32 +36,29 @@ from devil.utils import lsusb from devil.utils import usb_hubs with devil_env.SysPath(devil_env.PYMOCK_PATH): - import mock # pylint: disable=import-error + import mock # pylint: disable=import-error # Output of lsusb.lsusb(). # We just test that the dictionary is working by creating an # "ID number" equal to (bus_num*1000)+device_num and seeing if # it is picked up correctly. Also we test the description -DEVLIST = [(1, 11, 'foo'), - (1, 12, 'bar'), - (1, 13, 'baz'), - (2, 11, 'quux'), - (2, 20, 'My Test HUB'), - (2, 21, 'ID 0403:6001 usb_device_p7_h1_t0'), +DEVLIST = [(1, 11, 'foo'), (1, 12, 'bar'), (1, 13, 'baz'), (2, 11, 'quux'), + (2, 20, 'My Test HUB'), (2, 21, 'ID 0403:6001 usb_device_p7_h1_t0'), (2, 22, 'ID 0403:6001 usb_device_p5_h1_t1'), (2, 23, 'My Test Internal HUB'), (2, 24, 'ID 0403:6001 usb_device_p3_h1_t2'), (2, 25, 'ID 0403:6001 usb_device_p1_h1_t3'), - (2, 26, 'Not a Battery Monitor'), - (2, 100, 'My Test HUB'), + (2, 26, 'Not a Battery Monitor'), (2, 100, 'My Test HUB'), (2, 101, 'My Test Internal HUB'), (2, 102, 'ID 0403:6001 usb_device_p1_h1_t4')] -LSUSB_OUTPUT = [ - {'bus': b, 'device': d, 'desc': t, 'id': (1000*b)+d} - for (b, d, t) in DEVLIST] - +LSUSB_OUTPUT = [{ + 'bus': b, + 'device': d, + 'desc': t, + 'id': (1000 * b) + d +} for (b, d, t) in DEVLIST] # Note: "Lev", "Cnt", "Spd", and "MxCh" are not used by parser, # so we just leave them as zeros here. Also note that the port @@ -170,12 +166,14 @@ ATTRS{devnum}=="0" ''' UDEVADM_OUTPUT_DICT = { - 'ttyUSB0': UDEVADM_USBTTY0_OUTPUT, - 'ttyUSB1': UDEVADM_USBTTY1_OUTPUT, - 'ttyUSB2': UDEVADM_USBTTY2_OUTPUT, - 'ttyUSB3': UDEVADM_USBTTY3_OUTPUT, - 'ttyUSB4': UDEVADM_USBTTY4_OUTPUT, - 'ttyUSB5': UDEVADM_USBTTY5_OUTPUT} + 'ttyUSB0': UDEVADM_USBTTY0_OUTPUT, + 'ttyUSB1': UDEVADM_USBTTY1_OUTPUT, + 'ttyUSB2': UDEVADM_USBTTY2_OUTPUT, + 'ttyUSB3': UDEVADM_USBTTY3_OUTPUT, + 'ttyUSB4': UDEVADM_USBTTY4_OUTPUT, + 'ttyUSB5': UDEVADM_USBTTY5_OUTPUT +} + # Identification criteria for Plugable 7-Port Hub def isTestHub(node): @@ -192,11 +190,19 @@ def isTestHub(node): return False return 'Test Internal HUB' in node.PortToDevice(4).desc -TEST_HUB = usb_hubs.HubType(isTestHub, - {1:7, - 2:6, - 3:5, - 4:{1:4, 2:3, 3:2, 4:1}}) + +TEST_HUB = usb_hubs.HubType(isTestHub, { + 1: 7, + 2: 6, + 3: 5, + 4: { + 1: 4, + 2: 3, + 3: 2, + 4: 1 + } +}) + class USBScriptTest(unittest.TestCase): def setUp(self): @@ -206,38 +212,42 @@ class USBScriptTest(unittest.TestCase): return_value=LSUSB_OUTPUT) find_usb_devices._GetUSBDevicesOutput = mock.Mock( return_value=USB_DEVICES_OUTPUT) - find_usb_devices._GetCommList = mock.Mock( - return_value=LIST_TTY_OUTPUT) - lsusb.raw_lsusb = mock.Mock( - return_value=RAW_LSUSB_OUTPUT) + find_usb_devices._GetCommList = mock.Mock(return_value=LIST_TTY_OUTPUT) + lsusb.raw_lsusb = mock.Mock(return_value=RAW_LSUSB_OUTPUT) def testGetTTYDevices(self): pp = find_usb_devices.GetAllPhysicalPortToTTYMaps([TEST_HUB]) result = list(pp) - self.assertEquals(result[0], {7:'ttyUSB0', - 5:'ttyUSB1', - 3:'ttyUSB2', - 2:'ttyUSB5', - 1:'ttyUSB3'}) - self.assertEquals(result[1], {1:'ttyUSB4'}) + self.assertEquals(result[0], { + 7: 'ttyUSB0', + 5: 'ttyUSB1', + 3: 'ttyUSB2', + 2: 'ttyUSB5', + 1: 'ttyUSB3' + }) + self.assertEquals(result[1], {1: 'ttyUSB4'}) def testGetPortDeviceMapping(self): pp = find_usb_devices.GetAllPhysicalPortToBusDeviceMaps([TEST_HUB]) result = list(pp) - self.assertEquals(result[0], {7:(2, 21), - 5:(2, 22), - 3:(2, 24), - 2:(2, 26), - 1:(2, 25)}) - self.assertEquals(result[1], {1:(2, 102)}) + self.assertEquals(result[0], { + 7: (2, 21), + 5: (2, 22), + 3: (2, 24), + 2: (2, 26), + 1: (2, 25) + }) + self.assertEquals(result[1], {1: (2, 102)}) def testGetSerialMapping(self): pp = find_usb_devices.GetAllPhysicalPortToSerialMaps([TEST_HUB]) result = list(pp) - self.assertEquals(result[0], {7:'UsbDevice0', - 5:'UsbDevice1', - 3:'UsbDevice2', - 1:'UsbDevice3'}) + self.assertEquals(result[0], { + 7: 'UsbDevice0', + 5: 'UsbDevice1', + 3: 'UsbDevice2', + 1: 'UsbDevice3' + }) self.assertEquals(result[1], {}) def testFastDeviceDescriptions(self): @@ -248,7 +258,7 @@ class USBScriptTest(unittest.TestCase): self.assertEquals(dev_foo.desc, 'FAST foo') self.assertEquals(dev_bar.desc, 'FAST bar') self.assertEquals(dev_usb_device_p7_h1_t0.desc, - 'ID 0403:6001 usb_device_p7_h1_t0') + 'ID 0403:6001 usb_device_p7_h1_t0') def testDeviceDescriptions(self): bd = find_usb_devices.GetBusNumberToDeviceTreeMap(fast=False) @@ -258,7 +268,7 @@ class USBScriptTest(unittest.TestCase): self.assertEquals(dev_foo.desc, 'foo') self.assertEquals(dev_bar.desc, 'bar') self.assertEquals(dev_usb_device_p7_h1_t0.desc, - 'ID 0403:6001 usb_device_p7_h1_t0') + 'ID 0403:6001 usb_device_p7_h1_t0') def testDeviceInformation(self): bd = find_usb_devices.GetBusNumberToDeviceTreeMap(fast=False) diff --git a/catapult/devil/devil/utils/geometry.py b/catapult/devil/devil/utils/geometry.py index da21770b..6da6d034 100644 --- a/catapult/devil/devil/utils/geometry.py +++ b/catapult/devil/devil/utils/geometry.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Objects for convenient manipulation of points and other surface areas.""" import collections diff --git a/catapult/devil/devil/utils/geometry_test.py b/catapult/devil/devil/utils/geometry_test.py index af694429..e2024828 100644 --- a/catapult/devil/devil/utils/geometry_test.py +++ b/catapult/devil/devil/utils/geometry_test.py @@ -1,7 +1,6 @@ # Copyright 2015 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Tests for the geometry module.""" import unittest @@ -10,7 +9,6 @@ from devil.utils import geometry as g class PointTest(unittest.TestCase): - def testStr(self): p = g.Point(1, 2) self.assertEquals(str(p), '(1, 2)') @@ -45,7 +43,6 @@ class PointTest(unittest.TestCase): class RectangleTest(unittest.TestCase): - def testStr(self): r = g.Rectangle(g.Point(0, 1), g.Point(2, 3)) self.assertEquals(str(r), '[(0, 1), (2, 3)]') diff --git a/catapult/devil/devil/utils/host_utils.py b/catapult/devil/devil/utils/host_utils.py index 6c337cf7..2918e06d 100644 --- a/catapult/devil/devil/utils/host_utils.py +++ b/catapult/devil/devil/utils/host_utils.py @@ -8,6 +8,7 @@ import os def GetRecursiveDiskUsage(path): """Returns the disk usage in bytes of |path|. Similar to `du -sb |path|`.""" + def get_size(filepath): try: return os.path.getsize(filepath) @@ -18,6 +19,6 @@ def GetRecursiveDiskUsage(path): running_size = get_size(path) if os.path.isdir(path): for root, dirs, files in os.walk(path): - running_size += sum([get_size(os.path.join(root, f)) - for f in files + dirs]) + running_size += sum( + [get_size(os.path.join(root, f)) for f in files + dirs]) return running_size diff --git a/catapult/devil/devil/utils/lazy/weak_constant.py b/catapult/devil/devil/utils/lazy/weak_constant.py index 24ad9406..41101937 100644 --- a/catapult/devil/devil/utils/lazy/weak_constant.py +++ b/catapult/devil/devil/utils/lazy/weak_constant.py @@ -30,11 +30,11 @@ class WeakConstant(object): # We initialize the value on a separate thread to protect # from holding self._lock indefinitely in the event that # self._initializer hangs. - initializer_thread = reraiser_thread.ReraiserThread( - self._initializer) + initializer_thread = reraiser_thread.ReraiserThread(self._initializer) initializer_thread.start() timeout_retry.WaitFor( - lambda: initializer_thread.join(1) or not initializer_thread.isAlive(), + lambda: initializer_thread.join(1) or not initializer_thread. + isAlive(), wait_period=0) self._val = initializer_thread.GetReturnValue() self._initialized.set() diff --git a/catapult/devil/devil/utils/lazy/weak_constant_test.py b/catapult/devil/devil/utils/lazy/weak_constant_test.py index 643351d8..583fd07a 100644 --- a/catapult/devil/devil/utils/lazy/weak_constant_test.py +++ b/catapult/devil/devil/utils/lazy/weak_constant_test.py @@ -30,14 +30,11 @@ class DynamicSideEffect(object): class WeakConstantTest(unittest.TestCase): - def testUninitialized(self): """Ensure that the first read calls the initializer.""" initializer = mock.Mock(return_value='initializer called') test_constant = lazy.WeakConstant(initializer) - self.assertEquals( - 'initializer called', - test_constant.read()) + self.assertEquals('initializer called', test_constant.read()) initializer.assert_called_once_with() def testInitialized(self): @@ -46,23 +43,18 @@ class WeakConstantTest(unittest.TestCase): test_constant = lazy.WeakConstant(initializer) test_constant._initialized.set() test_constant._val = 'initializer not called' - self.assertEquals( - 'initializer not called', - test_constant.read()) + self.assertEquals('initializer not called', test_constant.read()) self.assertFalse(initializer.mock_calls) # assert not called def testFirstCallHangs(self): """Ensure that reading works even if the first initializer call hangs.""" - dyn = DynamicSideEffect([ - lambda: time.sleep(10), - lambda: 'second try worked!' - ]) + dyn = DynamicSideEffect( + [lambda: time.sleep(10), lambda: 'second try worked!']) initializer = mock.Mock(side_effect=dyn) test_constant = lazy.WeakConstant(initializer) - self.assertEquals( - 'second try worked!', - timeout_retry.Run(test_constant.read, 1, 1)) + self.assertEquals('second try worked!', + timeout_retry.Run(test_constant.read, 1, 1)) initializer.assert_has_calls([mock.call(), mock.call()]) diff --git a/catapult/devil/devil/utils/logging_common.py b/catapult/devil/devil/utils/logging_common.py index ab364a20..3df7aab6 100644 --- a/catapult/devil/devil/utils/logging_common.py +++ b/catapult/devil/devil/utils/logging_common.py @@ -15,10 +15,16 @@ def AddLoggingArguments(parser): """ group = parser.add_mutually_exclusive_group() group.add_argument( - '-v', '--verbose', action='count', default=0, + '-v', + '--verbose', + action='count', + default=0, help='Log more. Use multiple times for even more logging.') group.add_argument( - '-q', '--quiet', action='count', default=0, + '-q', + '--quiet', + action='count', + default=0, help=('Log less (suppress output). Use multiple times for even less ' 'output.')) @@ -66,4 +72,3 @@ class CustomFormatter(logging.Formatter): msg = msg.replace('MainThread', 'Main', 1) timediff = time.time() - self._creation_time return '%s %8.3fs %s' % (record.levelname[0], timediff, msg) - diff --git a/catapult/devil/devil/utils/lsusb.py b/catapult/devil/devil/utils/lsusb.py index 6cbf2567..dd5d471f 100644 --- a/catapult/devil/devil/utils/lsusb.py +++ b/catapult/devil/devil/utils/lsusb.py @@ -85,11 +85,9 @@ def _lsusbv_on_device(bus_id, dev_id): m = _LSUSB_BUS_DEVICE_RE.match(line) if m: if m.group(1) != bus_id: - logger.warning( - 'Expected bus_id value: %r, seen %r', bus_id, m.group(1)) + logger.warning('Expected bus_id value: %r, seen %r', bus_id, m.group(1)) if m.group(2) != dev_id: - logger.warning( - 'Expected dev_id value: %r, seen %r', dev_id, m.group(2)) + logger.warning('Expected dev_id value: %r, seen %r', dev_id, m.group(2)) device['desc'] = m.group(3) continue @@ -102,8 +100,7 @@ def _lsusbv_on_device(bus_id, dev_id): # Determine the indentation depth. depth = 1 + len(indent_match.group(1)) / 2 if depth > len(depth_stack): - logger.error( - 'lsusb parsing error: unexpected indentation: "%s"', line) + logger.error('lsusb parsing error: unexpected indentation: "%s"', line) continue # Pop everything off the depth stack that isn't a parent of @@ -123,8 +120,8 @@ def _lsusbv_on_device(bus_id, dev_id): m = _LSUSB_ENTRY_RE.match(line) if m: new_entry = { - '_value': m.group(2), - '_desc': m.group(3), + '_value': m.group(2), + '_desc': m.group(3), } cur[m.group(1)] = new_entry depth_stack.append(new_entry) @@ -134,10 +131,11 @@ def _lsusbv_on_device(bus_id, dev_id): return device + def lsusb(): """Call lsusb and return the parsed output.""" - _, lsusb_list_output = cmd_helper.GetCmdStatusAndOutputWithTimeout( - ['lsusb'], timeout=10) + _, lsusb_list_output = cmd_helper.GetCmdStatusAndOutputWithTimeout(['lsusb'], + timeout=10) devices = [] for line in lsusb_list_output.splitlines(): m = _LSUSB_BUS_DEVICE_RE.match(line) @@ -147,19 +145,22 @@ def lsusb(): try: devices.append(_lsusbv_on_device(bus_num, dev_num)) except cmd_helper.TimeoutError: - # Will be blacklisted if it is in expected device file, but times out. + # Will be denylisted if it is in expected device file, but times out. logger.info('lsusb -v %s:%s timed out.', bus_num, dev_num) return devices + def raw_lsusb(): return cmd_helper.GetCmdOutput(['lsusb']) + def get_lsusb_serial(device): try: return device['Device Descriptor']['iSerial']['_desc'] except KeyError: return None + def _is_android_device(device): try: # Hubs are not android devices. @@ -169,6 +170,7 @@ def _is_android_device(device): pass return get_lsusb_serial(device) is not None + def get_android_devices(): android_devices = (d for d in lsusb() if _is_android_device(d)) return [get_lsusb_serial(d) for d in android_devices] diff --git a/catapult/devil/devil/utils/lsusb_test.py b/catapult/devil/devil/utils/lsusb_test.py index f381e72f..6a870d1b 100755 --- a/catapult/devil/devil/utils/lsusb_test.py +++ b/catapult/devil/devil/utils/lsusb_test.py @@ -2,7 +2,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Tests for the cmd_helper module.""" import unittest @@ -12,7 +11,7 @@ from devil.utils import lsusb from devil.utils import mock_calls with devil_env.SysPath(devil_env.PYMOCK_PATH): - import mock # pylint: disable=import-error + import mock # pylint: disable=import-error RAW_OUTPUT = """ Bus 003 Device 007: ID 18d1:4ee2 Google Inc. Nexus 4 (debug) @@ -138,81 +137,210 @@ EXPECTED_RESULT = { } }, 'Device Descriptor': { - 'bLength': {'_value': '18', '_desc': None}, - 'bcdDevice': {'_value': '2.28', '_desc': None}, - 'bDeviceSubClass': {'_value': '0', '_desc': None}, - 'idVendor': {'_value': '0x18d1', '_desc': 'Google Inc.'}, - 'bcdUSB': {'_value': '2.00', '_desc': None}, - 'bDeviceProtocol': {'_value': '0', '_desc': None}, - 'bDescriptorType': {'_value': '1', '_desc': None}, + 'bLength': { + '_value': '18', + '_desc': None + }, + 'bcdDevice': { + '_value': '2.28', + '_desc': None + }, + 'bDeviceSubClass': { + '_value': '0', + '_desc': None + }, + 'idVendor': { + '_value': '0x18d1', + '_desc': 'Google Inc.' + }, + 'bcdUSB': { + '_value': '2.00', + '_desc': None + }, + 'bDeviceProtocol': { + '_value': '0', + '_desc': None + }, + 'bDescriptorType': { + '_value': '1', + '_desc': None + }, 'Configuration Descriptor': { - 'bLength': {'_value': '9', '_desc': None}, - 'wTotalLength': {'_value': '62', '_desc': None}, - 'bConfigurationValue': {'_value': '1', '_desc': None}, + 'bLength': { + '_value': '9', + '_desc': None + }, + 'wTotalLength': { + '_value': '62', + '_desc': None + }, + 'bConfigurationValue': { + '_value': '1', + '_desc': None + }, 'Interface Descriptor': { - 'bLength': {'_value': '9', '_desc': None}, - 'bAlternateSetting': {'_value': '0', '_desc': None}, - 'bInterfaceNumber': {'_value': '1', '_desc': None}, - 'bNumEndpoints': {'_value': '2', '_desc': None}, - 'bDescriptorType': {'_value': '4', '_desc': None}, - 'bInterfaceSubClass': {'_value': '66', '_desc': None}, + 'bLength': { + '_value': '9', + '_desc': None + }, + 'bAlternateSetting': { + '_value': '0', + '_desc': None + }, + 'bInterfaceNumber': { + '_value': '1', + '_desc': None + }, + 'bNumEndpoints': { + '_value': '2', + '_desc': None + }, + 'bDescriptorType': { + '_value': '4', + '_desc': None + }, + 'bInterfaceSubClass': { + '_value': '66', + '_desc': None + }, 'bInterfaceClass': { '_value': '255', '_desc': 'Vendor Specific Class' }, - 'bInterfaceProtocol': {'_value': '1', '_desc': None}, + 'bInterfaceProtocol': { + '_value': '1', + '_desc': None + }, 'Endpoint Descriptor': { - 'bLength': {'_value': '7', '_desc': None}, - 'bEndpointAddress': {'_value': '0x02', '_desc': 'EP 2 OUT'}, - 'bInterval': {'_value': '0', '_desc': None}, - 'bDescriptorType': {'_value': '5', '_desc': None}, + 'bLength': { + '_value': '7', + '_desc': None + }, + 'bEndpointAddress': { + '_value': '0x02', + '_desc': 'EP 2 OUT' + }, + 'bInterval': { + '_value': '0', + '_desc': None + }, + 'bDescriptorType': { + '_value': '5', + '_desc': None + }, 'bmAttributes': { '_value': '2', - 'Transfer': {'_value': 'Type', '_desc': 'Bulk'}, - 'Usage': {'_value': 'Type', '_desc': 'Data'}, + 'Transfer': { + '_value': 'Type', + '_desc': 'Bulk' + }, + 'Usage': { + '_value': 'Type', + '_desc': 'Data' + }, '_desc': None, - 'Synch': {'_value': 'Type', '_desc': 'None'} + 'Synch': { + '_value': 'Type', + '_desc': 'None' + } }, 'wMaxPacketSize': { '_value': '0x0040', '_desc': '1x 64 bytes' } }, - 'iInterface': {'_value': '0', '_desc': None} + 'iInterface': { + '_value': '0', + '_desc': None + } + }, + 'bDescriptorType': { + '_value': '2', + '_desc': None + }, + 'iConfiguration': { + '_value': '0', + '_desc': None }, - 'bDescriptorType': {'_value': '2', '_desc': None}, - 'iConfiguration': {'_value': '0', '_desc': None}, 'bmAttributes': { '_value': '0x80', '_desc': None, - '(Bus': {'_value': 'Powered)', '_desc': None} + '(Bus': { + '_value': 'Powered)', + '_desc': None + } + }, + 'bNumInterfaces': { + '_value': '2', + '_desc': None }, - 'bNumInterfaces': {'_value': '2', '_desc': None}, - 'MaxPower': {'_value': '500mA', '_desc': None} + 'MaxPower': { + '_value': '500mA', + '_desc': None + } + }, + 'iSerial': { + '_value': '3', + '_desc': '01d2450ea194a93b' + }, + 'idProduct': { + '_value': '0x4ee2', + '_desc': 'Nexus 4 (debug)' + }, + 'iManufacturer': { + '_value': '1', + '_desc': 'LGE' }, - 'iSerial': {'_value': '3', '_desc': '01d2450ea194a93b'}, - 'idProduct': {'_value': '0x4ee2', '_desc': 'Nexus 4 (debug)'}, - 'iManufacturer': {'_value': '1', '_desc': 'LGE'}, 'bDeviceClass': { '_value': '0', '_desc': '(Defined at Interface level)' }, - 'iProduct': {'_value': '2', '_desc': 'Nexus 4'}, - 'bMaxPacketSize0': {'_value': '64', '_desc': None}, - 'bNumConfigurations': {'_value': '1', '_desc': None} + 'iProduct': { + '_value': '2', + '_desc': 'Nexus 4' + }, + 'bMaxPacketSize0': { + '_value': '64', + '_desc': None + }, + 'bNumConfigurations': { + '_value': '1', + '_desc': None + } }, 'Device Qualifier (for other device speed)': { - 'bLength': {'_value': '10', '_desc': None}, - 'bNumConfigurations': {'_value': '1', '_desc': None}, - 'bDeviceSubClass': {'_value': '0', '_desc': None}, - 'bcdUSB': {'_value': '2.00', '_desc': None}, - 'bDeviceProtocol': {'_value': '0', '_desc': None}, - 'bDescriptorType': {'_value': '6', '_desc': None}, + 'bLength': { + '_value': '10', + '_desc': None + }, + 'bNumConfigurations': { + '_value': '1', + '_desc': None + }, + 'bDeviceSubClass': { + '_value': '0', + '_desc': None + }, + 'bcdUSB': { + '_value': '2.00', + '_desc': None + }, + 'bDeviceProtocol': { + '_value': '0', + '_desc': None + }, + 'bDescriptorType': { + '_value': '6', + '_desc': None + }, 'bDeviceClass': { '_value': '0', '_desc': '(Defined at Interface level)' }, - 'bMaxPacketSize0': {'_value': '64', '_desc': None} + 'bMaxPacketSize0': { + '_value': '64', + '_desc': None + } } } @@ -225,7 +353,7 @@ class LsusbTest(mock_calls.TestCase): (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutputWithTimeout( ['lsusb'], timeout=10), (None, DEVICE_LIST)), (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutputWithTimeout( - ['lsusb', '-v', '-s', '003:007'], timeout=10), (None, RAW_OUTPUT))): + ['lsusb', '-v', '-s', '003:007'], timeout=10), (None, RAW_OUTPUT))): self.assertDictEqual(lsusb.lsusb().pop(), EXPECTED_RESULT) def testGetSerial(self): @@ -233,7 +361,7 @@ class LsusbTest(mock_calls.TestCase): (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutputWithTimeout( ['lsusb'], timeout=10), (None, DEVICE_LIST)), (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutputWithTimeout( - ['lsusb', '-v', '-s', '003:007'], timeout=10), (None, RAW_OUTPUT))): + ['lsusb', '-v', '-s', '003:007'], timeout=10), (None, RAW_OUTPUT))): self.assertEqual(lsusb.get_android_devices(), ['01d2450ea194a93b']) def testGetLsusbSerial(self): @@ -241,7 +369,7 @@ class LsusbTest(mock_calls.TestCase): (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutputWithTimeout( ['lsusb'], timeout=10), (None, DEVICE_LIST)), (mock.call.devil.utils.cmd_helper.GetCmdStatusAndOutputWithTimeout( - ['lsusb', '-v', '-s', '003:007'], timeout=10), (None, RAW_OUTPUT))): + ['lsusb', '-v', '-s', '003:007'], timeout=10), (None, RAW_OUTPUT))): out = lsusb.lsusb().pop() self.assertEqual(lsusb.get_lsusb_serial(out), '01d2450ea194a93b') diff --git a/catapult/devil/devil/utils/markdown.py b/catapult/devil/devil/utils/markdown.py index ba666643..12d2eb3c 100755 --- a/catapult/devil/devil/utils/markdown.py +++ b/catapult/devil/devil/utils/markdown.py @@ -19,8 +19,8 @@ _CODE_BLOCK_FORMAT = '''```{language} ``` ''' -_DEVIL_ROOT = os.path.abspath(os.path.join( - os.path.dirname(__file__), '..', '..')) +_DEVIL_ROOT = os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..')) def md_bold(raw_text): @@ -31,14 +31,15 @@ def md_bold(raw_text): def md_code(raw_text, language): """Returns a markdown-formatted code block in the given language.""" return _CODE_BLOCK_FORMAT.format( - language=language or '', - code=md_escape(raw_text, characters='`')) + language=language or '', code=md_escape(raw_text, characters='`')) def md_escape(raw_text, characters='*_'): """Escapes * and _.""" + def escape_char(m): return '\\%s' % m.group(0) + pattern = '[%s]' % re.escape(characters) return re.sub(pattern, escape_char, raw_text) @@ -46,8 +47,8 @@ def md_escape(raw_text, characters='*_'): def md_heading(raw_text, level): """Returns markdown-formatted heading.""" adjusted_level = min(max(level, 0), 6) - return '%s%s%s' % ( - '#' * adjusted_level, ' ' if adjusted_level > 0 else '', raw_text) + return '%s%s%s' % ('#' * adjusted_level, ' ' if adjusted_level > 0 else '', + raw_text) def md_inline_code(raw_text): @@ -62,9 +63,8 @@ def md_italic(raw_text): def md_link(link_text, link_target): """returns a markdown-formatted link.""" - return '[%s](%s)' % ( - md_escape(link_text, characters=']'), - md_escape(link_target, characters=')')) + return '[%s](%s)' % (md_escape(link_text, characters=']'), + md_escape(link_target, characters=')')) class MarkdownHelpFormatter(argparse.HelpFormatter): @@ -112,8 +112,10 @@ class MarkdownHelpFormatter(argparse.HelpFormatter): class MarkdownHelpAction(argparse.Action): - def __init__(self, option_strings, - dest=argparse.SUPPRESS, default=argparse.SUPPRESS, + def __init__(self, + option_strings, + dest=argparse.SUPPRESS, + default=argparse.SUPPRESS, **kwargs): super(MarkdownHelpAction, self).__init__( option_strings=option_strings, @@ -137,8 +139,10 @@ def add_md_help_argument(parser): Args: parser: The ArgumentParser to which --md-help should be added. """ - parser.add_argument('--md-help', action=MarkdownHelpAction, - help='print Markdown-formatted help text and exit.') + parser.add_argument( + '--md-help', + action=MarkdownHelpAction, + help='print Markdown-formatted help text and exit.') def load_module_from_path(module_path): @@ -172,24 +176,25 @@ def load_module_from_path(module_path): return module -def md_module(module_obj, module_path=None, module_link=None): - """Write markdown documentation for a class. +def md_module(module_obj, module_link=None): + """Write markdown documentation for a module. Documents public classes and functions. Args: - class_obj: a types.TypeType object for the class that should be - documented. + module_obj: a module object that should be documented. Returns: A list of markdown-formatted lines. """ + def should_doc(name): return (not isinstance(module_obj.__dict__[name], types.ModuleType) and not name.startswith('_')) - stuff_to_doc = sorted( - obj for name, obj in module_obj.__dict__.iteritems() - if should_doc(name)) + stuff_to_doc = [ + obj for name, obj in sorted(module_obj.__dict__.iteritems()) + if should_doc(name) + ] classes_to_doc = [] functions_to_doc = [] @@ -200,12 +205,6 @@ def md_module(module_obj, module_path=None, module_link=None): elif isinstance(s, types.FunctionType): functions_to_doc.append(s) - command = ['devil/utils/markdown.py'] - if module_link: - command.extend(['--module-link', module_link]) - if module_path: - command.append(os.path.relpath(module_path, _DEVIL_ROOT)) - heading_text = module_obj.__name__ if module_link: heading_text = md_link(heading_text, module_link) @@ -213,8 +212,8 @@ def md_module(module_obj, module_path=None, module_link=None): content = [ md_heading(heading_text, level=1), '', - md_italic('This page was autogenerated by %s' - % md_inline_code(' '.join(command))), + md_italic('This page was autogenerated. ' + 'Run `devil/bin/generate_md_docs` to update'), '', ] @@ -248,9 +247,10 @@ def md_class(class_obj): return (isinstance(obj, types.FunctionType) and (name.startswith('__') or not name.startswith('_'))) - methods_to_doc = sorted( - obj for name, obj in class_obj.__dict__.iteritems() - if should_doc(name, obj)) + methods_to_doc = [ + obj for name, obj in sorted(class_obj.__dict__.iteritems()) + if should_doc(name, obj) + ] for m in methods_to_doc: content.extend(md_function(m, class_obj=class_obj)) @@ -313,10 +313,8 @@ def main(raw_args): args = parser.parse_args(raw_args) return md_module( - load_module_from_path(args.module_path), - module_link=args.module_link) + load_module_from_path(args.module_path), module_link=args.module_link) if __name__ == '__main__': sys.exit(main(sys.argv[1:])) - diff --git a/catapult/devil/devil/utils/markdown_test.py b/catapult/devil/devil/utils/markdown_test.py index 323776ca..621d56ba 100755 --- a/catapult/devil/devil/utils/markdown_test.py +++ b/catapult/devil/devil/utils/markdown_test.py @@ -16,7 +16,6 @@ from devil.utils import markdown class MarkdownTest(unittest.TestCase): - def testBold(self): raw = 'foo' self.assertEquals('**foo**', markdown.md_bold(raw)) @@ -85,13 +84,13 @@ class MarkdownTest(unittest.TestCase): def testInlineCode(self): raw = 'devil.utils.markdown_test' - self.assertEquals( - '`devil.utils.markdown_test`', markdown.md_inline_code(raw)) + self.assertEquals('`devil.utils.markdown_test`', + markdown.md_inline_code(raw)) def testInlineCodeContainsTicks(self): raw = 'this contains `backticks`' - self.assertEquals( - '`this contains \\`backticks\\``', markdown.md_inline_code(raw)) + self.assertEquals('`this contains \\`backticks\\``', + markdown.md_inline_code(raw)) def testItalic(self): raw = 'bar' diff --git a/catapult/devil/devil/utils/mock_calls.py b/catapult/devil/devil/utils/mock_calls.py index 5ae951e3..2b359385 100644 --- a/catapult/devil/devil/utils/mock_calls.py +++ b/catapult/devil/devil/utils/mock_calls.py @@ -1,7 +1,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ A test facility to assert call sequences while mocking their behavior. """ @@ -16,8 +15,8 @@ with devil_env.SysPath(devil_env.PYMOCK_PATH): class TestCase(unittest.TestCase): """Adds assertCalls to TestCase objects.""" - class _AssertCalls(object): + class _AssertCalls(object): def __init__(self, test_case, expected_calls, watched): def call_action(pair): if isinstance(pair, type(mock.call)): @@ -36,21 +35,24 @@ class TestCase(unittest.TestCase): received_call == expected_call, msg=('Expected call mismatch:\n' ' expected: %s\n' - ' received: %s\n' - % (str(expected_call), str(received_call)))) + ' received: %s\n' % (str(expected_call), + str(received_call)))) if callable(action): return action(*args, **kwargs) else: return action + return side_effect self._test_case = test_case self._expected_calls = [call_action(pair) for pair in expected_calls] watched = watched.copy() # do not pollute the caller's dict - watched.update((call.parent.name, call.parent) - for call, _ in self._expected_calls) - self._patched = [test_case.patch_call(call, side_effect=do_check(call)) - for call in watched.itervalues()] + watched.update( + (call.parent.name, call.parent) for call, _ in self._expected_calls) + self._patched = [ + test_case.patch_call(call, side_effect=do_check(call)) + for call in watched.itervalues() + ] def __enter__(self): for patch in self._patched: @@ -61,11 +63,10 @@ class TestCase(unittest.TestCase): for patch in self._patched: patch.__exit__(exc_type, exc_val, exc_tb) if exc_type is None: - missing = ''.join(' expected: %s\n' % str(call) - for call, _ in self._expected_calls) + missing = ''.join( + ' expected: %s\n' % str(call) for call, _ in self._expected_calls) self._test_case.assertFalse( - missing, - msg='Expected calls not found:\n' + missing) + missing, msg='Expected calls not found:\n' + missing) def __init__(self, *args, **kwargs): super(TestCase, self).__init__(*args, **kwargs) @@ -135,9 +136,9 @@ class TestCase(unittest.TestCase): target = self.call_target(call) if ignore is None: ignore = [] - self.watchCalls(getattr(call, method) - for method in dir(target.__class__) - if not method.startswith('_') and not method in ignore) + self.watchCalls( + getattr(call, method) for method in dir(target.__class__) + if not method.startswith('_') and not method in ignore) def clearWatched(self): """Clear the set of watched calls.""" @@ -177,4 +178,3 @@ class TestCase(unittest.TestCase): def assertCall(self, call, action=None): return self.assertCalls((call, action)) - diff --git a/catapult/devil/devil/utils/mock_calls_test.py b/catapult/devil/devil/utils/mock_calls_test.py index 8eb4fc9d..f1e71780 100755 --- a/catapult/devil/devil/utils/mock_calls_test.py +++ b/catapult/devil/devil/utils/mock_calls_test.py @@ -2,7 +2,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Unit tests for the contents of mock_calls.py. """ @@ -20,7 +19,6 @@ with devil_env.SysPath(devil_env.PYMOCK_PATH): class _DummyAdb(object): - def __str__(self): return '0123456789abcdef' @@ -45,13 +43,13 @@ class _DummyAdb(object): class TestCaseWithAssertCallsTest(mock_calls.TestCase): - def setUp(self): self.adb = _DummyAdb() def ShellError(self): def action(cmd): raise ValueError('(device %s) command %r is not nice' % (self.adb, cmd)) + return action def get_answer(self): @@ -63,8 +61,7 @@ class TestCaseWithAssertCallsTest(mock_calls.TestCase): return thing def testCallTarget_succeds(self): - self.assertEquals(self.adb.Shell, - self.call_target(self.call.adb.Shell)) + self.assertEquals(self.adb.Shell, self.call_target(self.call.adb.Shell)) def testCallTarget_failsExternal(self): with self.assertRaises(ValueError): @@ -100,8 +97,7 @@ class TestCaseWithAssertCallsTest(mock_calls.TestCase): def testPatchCall_property(self): self.assertEquals(version_codes.LOLLIPOP, self.adb.build_version_sdk) with self.patch_call( - self.call.adb.build_version_sdk, - return_value=version_codes.KITKAT): + self.call.adb.build_version_sdk, return_value=version_codes.KITKAT): self.assertEquals(version_codes.KITKAT, self.adb.build_version_sdk) self.assertEquals(version_codes.LOLLIPOP, self.adb.build_version_sdk) @@ -114,10 +110,9 @@ class TestCaseWithAssertCallsTest(mock_calls.TestCase): def testAssertCalls_succeeds_multiple(self): with self.assertCalls( (mock.call.os.getcwd(), '/some/path'), - (self.call.echo('hello'), 'hello'), - (self.call.get_answer(), 11), - self.call.adb.Push('this_file', 'that_file'), - (self.call.get_answer(), 12)): + (self.call.echo('hello'), 'hello'), (self.call.get_answer(), 11), + self.call.adb.Push('this_file', + 'that_file'), (self.call.get_answer(), 12)): self.assertEquals(os.getcwd(), '/some/path') self.assertEquals('hello', self.echo('hello')) self.assertEquals(11, self.get_answer()) @@ -125,8 +120,7 @@ class TestCaseWithAssertCallsTest(mock_calls.TestCase): self.assertEquals(12, self.get_answer()) def testAsserCalls_succeeds_withAction(self): - with self.assertCall( - self.call.adb.Shell('echo hello'), self.ShellError()): + with self.assertCall(self.call.adb.Shell('echo hello'), self.ShellError()): with self.assertRaises(ValueError): self.adb.Shell('echo hello') @@ -170,4 +164,3 @@ class TestCaseWithAssertCallsTest(mock_calls.TestCase): if __name__ == '__main__': logging.getLogger().setLevel(logging.DEBUG) unittest.main(verbosity=2) - diff --git a/catapult/devil/devil/utils/parallelizer.py b/catapult/devil/devil/utils/parallelizer.py index 678066c7..930d01f9 100644 --- a/catapult/devil/devil/utils/parallelizer.py +++ b/catapult/devil/devil/utils/parallelizer.py @@ -1,7 +1,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """ Wrapper that allows method execution in parallel. This class wraps a list of objects of the same type, emulates their @@ -113,11 +112,11 @@ class Parallelizer(object): raise AttributeError("'%s' is not callable" % o.__name__) r = type(self)(self._orig_objs) - r._objs = reraiser_thread.ReraiserThreadGroup( - [reraiser_thread.ReraiserThread( - o, args=args, kwargs=kwargs, - name='%s.%s' % (str(d), o.__name__)) - for d, o in zip(self._orig_objs, self._objs)]) + r._objs = reraiser_thread.ReraiserThreadGroup([ + reraiser_thread.ReraiserThread( + o, args=args, kwargs=kwargs, name='%s.%s' % (str(d), o.__name__)) + for d, o in zip(self._orig_objs, self._objs) + ]) r._objs.StartAll() return r @@ -169,11 +168,14 @@ class Parallelizer(object): """ self._assertNoShadow('pMap') r = type(self)(self._orig_objs) - r._objs = reraiser_thread.ReraiserThreadGroup( - [reraiser_thread.ReraiserThread( - f, args=tuple([o] + list(args)), kwargs=kwargs, + r._objs = reraiser_thread.ReraiserThreadGroup([ + reraiser_thread.ReraiserThread( + f, + args=tuple([o] + list(args)), + kwargs=kwargs, name='%s(%s)' % (f.__name__, d)) - for d, o in zip(self._orig_objs, self._objs)]) + for d, o in zip(self._orig_objs, self._objs) + ]) r._objs.StartAll() return r @@ -262,4 +264,3 @@ class SyncParallelizer(Parallelizer): r = super(SyncParallelizer, self).pMap(f, *args, **kwargs) r.pFinish(None) return r - diff --git a/catapult/devil/devil/utils/parallelizer_test.py b/catapult/devil/devil/utils/parallelizer_test.py index acbb986e..7c86148c 100644 --- a/catapult/devil/devil/utils/parallelizer_test.py +++ b/catapult/devil/devil/utils/parallelizer_test.py @@ -2,7 +2,6 @@ # Copyright 2014 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Unit tests for the contents of parallelizer.py.""" # pylint: disable=protected-access @@ -16,8 +15,8 @@ import sys import unittest if __name__ == '__main__': - sys.path.append(os.path.abspath( - os.path.join(os.path.dirname(__file__), '..', '..'))) + sys.path.append( + os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))) from devil.utils import parallelizer @@ -69,7 +68,6 @@ class ParallelizerTestObject(object): class ParallelizerTestObjectHelper(object): - def __init__(self, thing): self._thing = thing @@ -78,7 +76,6 @@ class ParallelizerTestObjectHelper(object): class ParallelizerTest(unittest.TestCase): - def testInitEmptyList(self): r = parallelizer.Parallelizer([]).replace('a', 'b').pGet(0.1) self.assertEquals([], r) @@ -97,15 +94,17 @@ class ParallelizerTest(unittest.TestCase): def testAllReturn(self): devices = [ParallelizerTestObject(True) for _ in xrange(0, 10)] - results = ParallelizerTestObject.parallel( - devices).doReturnTheThing().pGet(1) + results = ParallelizerTestObject.parallel(devices).doReturnTheThing().pGet( + 1) self.assertTrue(isinstance(results, list)) self.assertEquals(10, len(results)) self.assertTrue(all(results)) def testAllRaise(self): - devices = [ParallelizerTestObject(Exception('thing %d' % i)) - for i in xrange(0, 10)] + devices = [ + ParallelizerTestObject(Exception('thing %d' % i)) + for i in xrange(0, 10) + ] p = ParallelizerTestObject.parallel(devices).doRaiseTheThing() with self.assertRaises(Exception): p.pGet(1) @@ -116,13 +115,16 @@ class ParallelizerTest(unittest.TestCase): exception_msg = 'thing %d' % exception_index try: - completion_files = [tempfile.NamedTemporaryFile(delete=False) - for _ in xrange(0, parallel_device_count)] + completion_files = [ + tempfile.NamedTemporaryFile(delete=False) + for _ in xrange(0, parallel_device_count) + ] devices = [ ParallelizerTestObject( i if i != exception_index else Exception(exception_msg), completion_files[i].name) - for i in xrange(0, parallel_device_count)] + for i in xrange(0, parallel_device_count) + ] for f in completion_files: f.close() p = ParallelizerTestObject.parallel(devices) @@ -151,8 +153,8 @@ class ParallelizerTest(unittest.TestCase): def testContained(self): devices = [ParallelizerTestObject(i) for i in xrange(0, 10)] - results = (ParallelizerTestObject.parallel(devices).helper - .doReturnStringThing().pGet(1)) + results = (ParallelizerTestObject.parallel(devices).helper. + doReturnStringThing().pGet(1)) self.assertTrue(isinstance(results, list)) self.assertEquals(10, len(results)) for i in xrange(0, 10): @@ -165,7 +167,6 @@ class ParallelizerTest(unittest.TestCase): class SyncParallelizerTest(unittest.TestCase): - def testContextManager(self): in_context = [False for i in xrange(10)] @@ -187,4 +188,3 @@ class SyncParallelizerTest(unittest.TestCase): if __name__ == '__main__': unittest.main(verbosity=2) - diff --git a/catapult/devil/devil/utils/reraiser_thread.py b/catapult/devil/devil/utils/reraiser_thread.py index 6e6c810b..e48dda92 100644 --- a/catapult/devil/devil/utils/reraiser_thread.py +++ b/catapult/devil/devil/utils/reraiser_thread.py @@ -1,7 +1,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Thread and ThreadGroup that reraise exceptions on the main thread.""" # pylint: disable=W0212 @@ -17,6 +16,7 @@ from devil.utils import watchdog_timer class TimeoutError(base_error.BaseError): """Module-specific timeout exception.""" + def __init__(self, message): super(TimeoutError, self).__init__(message) @@ -69,13 +69,14 @@ class ReraiserThread(threading.Thread): self._exc_info = None self._thread_group = None - if sys.version_info < (3,): + if sys.version_info < (3, ): # pylint: disable=exec-used - exec('''def ReraiseIfException(self): + exec ('''def ReraiseIfException(self): """Reraise exception if an exception was raised in the thread.""" if self._exc_info: raise self._exc_info[0], self._exc_info[1], self._exc_info[2]''') else: + def ReraiseIfException(self): """Reraise exception if an exception was raised in the thread.""" if self._exc_info: diff --git a/catapult/devil/devil/utils/reraiser_thread_unittest.py b/catapult/devil/devil/utils/reraiser_thread_unittest.py index e3c4e6be..eb37456c 100644 --- a/catapult/devil/devil/utils/reraiser_thread_unittest.py +++ b/catapult/devil/devil/utils/reraiser_thread_unittest.py @@ -1,7 +1,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Unittests for reraiser_thread.py.""" import threading @@ -52,7 +51,7 @@ class TestReraiserThreadGroup(unittest.TestCase): ran[i] = True group = reraiser_thread.ReraiserThreadGroup( - [reraiser_thread.ReraiserThread(f, args=[i]) for i in range(5)]) + [reraiser_thread.ReraiserThread(f, args=[i]) for i in range(5)]) group.StartAll() group.JoinAll() for v in ran: @@ -75,8 +74,9 @@ class TestReraiserThreadGroup(unittest.TestCase): def testJoinRaise(self): def f(): raise TestException + group = reraiser_thread.ReraiserThreadGroup( - [reraiser_thread.ReraiserThread(f) for _ in xrange(5)]) + [reraiser_thread.ReraiserThread(f) for _ in xrange(5)]) group.StartAll() with self.assertRaises(TestException): group.JoinAll() @@ -84,10 +84,12 @@ class TestReraiserThreadGroup(unittest.TestCase): def testJoinTimeout(self): def f(): pass + event = threading.Event() def g(): event.wait() + group = reraiser_thread.ReraiserThreadGroup( [reraiser_thread.ReraiserThread(g), reraiser_thread.ReraiserThread(f)]) @@ -113,5 +115,6 @@ class TestRunAsync(unittest.TestCase): self.assertEqual(1, a) self.assertEqual(2, b) + if __name__ == '__main__': unittest.main() diff --git a/catapult/devil/devil/utils/reset_usb.py b/catapult/devil/devil/utils/reset_usb.py index 404a44c6..d2f80b07 100755 --- a/catapult/devil/devil/utils/reset_usb.py +++ b/catapult/devil/devil/utils/reset_usb.py @@ -15,8 +15,7 @@ import re if __name__ == '__main__': sys.path.append( - os.path.abspath(os.path.join(os.path.dirname(__file__), - '..', '..'))) + os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))) from devil.android import device_errors from devil.utils import lsusb @@ -56,8 +55,8 @@ def reset_android_usb(serial): reset_usb(bus, device) else: raise device_errors.DeviceUnreachableError( - 'Unable to determine bus(%s) or device(%s) for device %s' - % (bus, device, serial)) + 'Unable to determine bus(%s) or device(%s) for device %s' % + (bus, device, serial)) def reset_all_android_devices(): @@ -75,17 +74,13 @@ def _reset_all_matching(condition): reset_usb(bus, device) serial = lsusb.get_lsusb_serial(device_info) if serial: - logger.info( - 'Reset USB device (bus: %03d, device: %03d, serial: %s)', - bus, device, serial) + logger.info('Reset USB device (bus: %03d, device: %03d, serial: %s)', + bus, device, serial) else: - logger.info( - 'Reset USB device (bus: %03d, device: %03d)', - bus, device) + logger.info('Reset USB device (bus: %03d, device: %03d)', bus, device) except IOError: - logger.error( - 'Failed to reset USB device (bus: %03d, device: %03d)', - bus, device) + logger.error('Failed to reset USB device (bus: %03d, device: %03d)', + bus, device) def main(): @@ -111,4 +106,3 @@ def main(): if __name__ == '__main__': sys.exit(main()) - diff --git a/catapult/devil/devil/utils/run_tests_helper.py b/catapult/devil/devil/utils/run_tests_helper.py index 0b9dd47f..b161fd74 100644 --- a/catapult/devil/devil/utils/run_tests_helper.py +++ b/catapult/devil/devil/utils/run_tests_helper.py @@ -1,7 +1,6 @@ # Copyright (c) 2012 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Helper functions common to native, java and host-driven test runners.""" import collections @@ -9,12 +8,10 @@ import logging from devil.utils import logging_common - CustomFormatter = logging_common.CustomFormatter - -_WrappedLoggingArgs = collections.namedtuple( - '_WrappedLoggingArgs', ['verbose', 'quiet']) +_WrappedLoggingArgs = collections.namedtuple('_WrappedLoggingArgs', + ['verbose', 'quiet']) def SetLogLevel(verbose_count, add_handler=True): diff --git a/catapult/devil/devil/utils/timeout_retry.py b/catapult/devil/devil/utils/timeout_retry.py index d662c1d2..8b26ac70 100644 --- a/catapult/devil/devil/utils/timeout_retry.py +++ b/catapult/devil/devil/utils/timeout_retry.py @@ -1,7 +1,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """A utility to run functions with timeouts and retries.""" # pylint: disable=W0702 @@ -16,7 +15,6 @@ logger = logging.getLogger(__name__) class TimeoutRetryThreadGroup(reraiser_thread.ReraiserThreadGroup): - def __init__(self, timeout, threads=None): super(TimeoutRetryThreadGroup, self).__init__(threads) self._watcher = watchdog_timer.WatchdogTimer(timeout) @@ -107,8 +105,8 @@ def WaitFor(condition, wait_period=5, max_tries=None): return result if timeout_thread_group: # pylint: disable=no-member - timeout_thread_group.GetRemainingTime(wait_period, - suffix=' waiting for condition %r' % condition_name) + timeout_thread_group.GetRemainingTime( + wait_period, suffix=' waiting for condition %r' % condition_name) if wait_period: time.sleep(wait_period) return None @@ -118,8 +116,14 @@ def AlwaysRetry(_exception): return True -def Run(func, timeout, retries, args=None, kwargs=None, desc=None, - error_log_func=logging.critical, retry_if_func=AlwaysRetry): +def Run(func, + timeout, + retries, + args=None, + kwargs=None, + desc=None, + error_log_func=logging.critical, + retry_if_func=AlwaysRetry): """Runs the passed function in a separate thread with timeouts and retries. Args: @@ -148,14 +152,16 @@ def Run(func, timeout, retries, args=None, kwargs=None, desc=None, while True: thread_name = 'TimeoutThread-%d-for-%s' % (num_try, threading.current_thread().name) - child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), - name=thread_name) + child_thread = reraiser_thread.ReraiserThread( + lambda: func(*args, **kwargs), name=thread_name) try: thread_group = TimeoutRetryThreadGroup(timeout, threads=[child_thread]) thread_group.StartAll(will_block=True) while True: - thread_group.JoinAll(watcher=thread_group.GetWatcher(), timeout=60, - error_log_func=error_log_func) + thread_group.JoinAll( + watcher=thread_group.GetWatcher(), + timeout=60, + error_log_func=error_log_func) if thread_group.IsAlive(): logger.info('Still working on %s', desc) else: @@ -168,7 +174,6 @@ def Run(func, timeout, retries, args=None, kwargs=None, desc=None, except Exception as e: # pylint: disable=broad-except if num_try > retries or not retry_if_func(e): raise - error_log_func( - '(%s) Exception on %s, attempt %d of %d: %r', - thread_name, desc, num_try, retries + 1, e) + error_log_func('(%s) Exception on %s, attempt %d of %d: %r', thread_name, + desc, num_try, retries + 1, e) num_try += 1 diff --git a/catapult/devil/devil/utils/timeout_retry_unittest.py b/catapult/devil/devil/utils/timeout_retry_unittest.py index 0eeb31a4..cf97bb9b 100755 --- a/catapult/devil/devil/utils/timeout_retry_unittest.py +++ b/catapult/devil/devil/utils/timeout_retry_unittest.py @@ -2,7 +2,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """Unittests for timeout_and_retry.py.""" import logging @@ -12,7 +11,6 @@ import unittest from devil.utils import reraiser_thread from devil.utils import timeout_retry - _DEFAULT_TIMEOUT = .1 @@ -29,8 +27,7 @@ class TestRun(unittest.TestCase): """Tests for timeout_retry.Run.""" def testRun(self): - self.assertTrue(timeout_retry.Run( - lambda x: x, 30, 3, [True], {})) + self.assertTrue(timeout_retry.Run(lambda x: x, 30, 3, [True], {})) def testTimeout(self): tries = [0] @@ -40,22 +37,34 @@ class TestRun(unittest.TestCase): time.sleep(1) self.assertRaises( - reraiser_thread.TimeoutError, timeout_retry.Run, _sleep, .01, 1, + reraiser_thread.TimeoutError, + timeout_retry.Run, + _sleep, + .01, + 1, error_log_func=logging.debug) self.assertEqual(tries[0], 2) def testRetries(self): tries = [0] self.assertRaises( - TestException, timeout_retry.Run, lambda: _CountTries(tries), - _DEFAULT_TIMEOUT, 3, error_log_func=logging.debug) + TestException, + timeout_retry.Run, + lambda: _CountTries(tries), + _DEFAULT_TIMEOUT, + 3, + error_log_func=logging.debug) self.assertEqual(tries[0], 4) def testNoRetries(self): tries = [0] self.assertRaises( - TestException, timeout_retry.Run, lambda: _CountTries(tries), - _DEFAULT_TIMEOUT, 0, error_log_func=logging.debug) + TestException, + timeout_retry.Run, + lambda: _CountTries(tries), + _DEFAULT_TIMEOUT, + 0, + error_log_func=logging.debug) self.assertEqual(tries[0], 1) def testReturnValue(self): @@ -70,7 +79,8 @@ class TestRun(unittest.TestCase): self.assertEqual(current_thread_group, timeout_retry.CurrentTimeoutThreadGroup()) return True - return reraiser_thread.RunAsync((InnerInnerFunc,))[0] + + return reraiser_thread.RunAsync((InnerInnerFunc, ))[0] self.assertTrue(timeout_retry.Run(InnerFunc, _DEFAULT_TIMEOUT, 3)) diff --git a/catapult/devil/devil/utils/update_dependencies.py b/catapult/devil/devil/utils/update_dependencies.py new file mode 100755 index 00000000..df1f88b2 --- /dev/null +++ b/catapult/devil/devil/utils/update_dependencies.py @@ -0,0 +1,218 @@ +#! /usr/bin/env python +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Updates the chromium binaries used by devil. + +This currently must be called from the top-level chromium src directory. +""" + +import argparse +import collections +import json +import logging +import os +import sys + +_DEVIL_ROOT_DIR = os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..')) + +sys.path.append(_DEVIL_ROOT_DIR) +from devil import base_error +from devil import devil_env +from devil.utils import cmd_helper + +_DEVICE_ARCHS = [ + { + 'cpu': 'arm', + 'platform': 'android_armeabi-v7a', + }, + { + 'cpu': 'arm64', + 'platform': 'android_arm64-v8a', + }, + { + 'cpu': 'x86', + 'platform': 'android_x86', + }, + { + 'cpu': 'x64', + 'platform': 'android_x86_64', + }, +] +_HOST_ARCH = [{ + # Host binaries use x86_64, not arm, but they build with the + # host toolchain within a target_cpu="arm" build. + 'cpu': 'arm', + 'platform': 'linux2_x86_64', +}] + +_CHROMIUM_DEPS = { + 'chromium_commands': { + 'archs': _HOST_ARCH, + 'build_path': 'lib.java/chromium_commands.dex.jar', + 'target_name': 'chromium_commands_java', + }, + 'forwarder_device': { + 'archs': _DEVICE_ARCHS, + 'build_path': 'device_forwarder', + 'target_name': 'forwarder2', + }, + 'forwarder_host': { + 'archs': _HOST_ARCH, + 'build_path': 'clang_x64/host_forwarder', + 'target_name': 'forwarder2', + }, + 'md5sum_device': { + 'archs': _DEVICE_ARCHS, + 'build_path': 'md5sum_bin', + 'target_name': 'md5sum', + }, + 'md5sum_host': { + 'archs': _HOST_ARCH, + 'build_path': 'clang_x64/md5sum_bin', + 'target_name': 'md5sum', + }, +} + + +def BuildTargetsForCpu(targets, cpu, output_dir): + logging.info('Building %s', cpu) + + gn_args = [ + 'ffmpeg_branding="Chrome"', + 'is_component_build=false', + 'is_debug=false', + 'proprietary_codecs=true', + 'symbol_level=1', + 'target_cpu="%s"' % cpu, + 'target_os="android"', + 'use_goma=true', + ] + + cmd = ['gn', 'gen', '--args=%s' % (' '.join(gn_args)), output_dir] + ec = cmd_helper.RunCmd(cmd) + if ec: + raise base_error.BaseError('%s failed with %d' % (cmd, ec)) + + ec = cmd_helper.RunCmd(['autoninja', '-C', output_dir] + targets) + if ec: + raise base_error.BaseError('building %s failed with %d' % (cpu, ec)) + + +def UpdateDependency(dependency_name, dependency_info, local_path, platform): + bucket = dependency_info['cloud_storage_bucket'] + folder = dependency_info['cloud_storage_base_folder'] + + # determine the hash + ec, sha1sum_output = cmd_helper.GetCmdStatusAndOutput(['sha1sum', local_path]) + if ec: + raise base_error.BaseError( + 'Failed to determine SHA1 for %s: %s' % (local_path, sha1sum_output)) + + dependency_sha1 = sha1sum_output.split()[0] + + # upload + remote_path = '%s_%s' % (dependency_name, dependency_sha1) + gs_dest = 'gs://%s/%s/%s' % (bucket, folder, remote_path) + ec, gsutil_output = cmd_helper.GetCmdStatusAndOutput( + ['gsutil', 'cp', local_path, gs_dest]) + if ec: + raise base_error.BaseError( + 'Failed to upload %s to %s: %s' % (remote_path, gs_dest, gsutil_output)) + + # update entry in json + file_info = dependency_info['file_info'] + if platform not in file_info: + file_info[platform] = { + 'cloud_storage_hash': '', + # the user will need to manually update the download path after + # uploading a previously unknown dependency. + 'download_path': 'FIXME', + } + file_info[platform]['cloud_storage_hash'] = dependency_sha1 + + +def UpdateChromiumDependencies(dependencies, args): + deps_by_platform = collections.defaultdict(list) + for dep_name, dep_info in _CHROMIUM_DEPS.iteritems(): + archs = dep_info.get('archs', []) + for a in archs: + deps_by_platform[(a.get('cpu'), a.get('platform'))].append( + (dep_name, dep_info.get('build_path'), dep_info.get('target_name'))) + + for arch, arch_deps in deps_by_platform.iteritems(): + targets = [target_name for _n, _b, target_name in arch_deps] + cpu, platform = arch + output_dir = os.path.join(args.chromium_src_dir, 'out-devil-deps', platform) + BuildTargetsForCpu(targets, cpu, output_dir) + + for dep_name, build_path, _ in arch_deps: + local_path = os.path.abspath(os.path.join(output_dir, build_path)) + UpdateDependency(dep_name, + dependencies.get('dependencies', {}).get(dep_name, {}), + local_path, platform) + + return dependencies + + +def UpdateGivenDependency(dependencies, args): + dep_name = args.name or os.path.basename(args.path) + if not dep_name in dependencies.get('dependencies', {}): + raise base_error.BaseError('Could not find dependency "%s" in %s' % + (dep_name, args.dependencies_json)) + + UpdateDependency(dep_name, + dependencies.get('dependencies', {}).get(dep_name, {}), + args.path, args.platform) + + return dependencies + + +def main(raw_args): + parser = argparse.ArgumentParser(description=__doc__) + + # pylint: disable=protected-access + parser.add_argument( + '--dependencies-json', + type=os.path.abspath, + default=devil_env._DEVIL_DEFAULT_CONFIG, + help='Binary dependency configuration file to update.') + # pylint: enable=protected-access + + subparsers = parser.add_subparsers() + chromium_parser = subparsers.add_parser('chromium') + chromium_parser.add_argument( + '--chromium-src-dir', + type=os.path.realpath, + default=os.getcwd(), + help='Path to chromium/src checkout root.') + chromium_parser.set_defaults(update_dependencies=UpdateChromiumDependencies) + + dependency_parser = subparsers.add_parser('dependency') + dependency_parser.add_argument('--name', help='Name of dependency to update.') + dependency_parser.add_argument( + '--path', + type=os.path.abspath, + help='Path to file to upload as new version of dependency.') + dependency_parser.add_argument( + '--platform', help='Platform of dependency to update.') + dependency_parser.set_defaults(update_dependencies=UpdateGivenDependency) + + args = parser.parse_args(raw_args) + + logging.getLogger().setLevel(logging.INFO) + + with open(args.dependencies_json) as f: + dependencies = json.load(f) + + dependencies = args.update_dependencies(dependencies, args) + + with open(args.dependencies_json, 'w') as f: + json.dump(dependencies, f, indent=2, separators=(',', ': '), sort_keys=True) + + return 0 + + +if __name__ == '__main__': + sys.exit(main(sys.argv[1:])) diff --git a/catapult/devil/devil/utils/usb_hubs.py b/catapult/devil/devil/utils/usb_hubs.py index bd984c7b..313cf3f6 100644 --- a/catapult/devil/devil/utils/usb_hubs.py +++ b/catapult/devil/devil/utils/usb_hubs.py @@ -2,25 +2,14 @@ # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -PLUGABLE_7PORT_LAYOUT = {1:7, - 2:6, - 3:5, - 4:{1:4, 2:3, 3:2, 4:1}} - -PLUGABLE_7PORT_USB3_LAYOUT = {1:{1:1, 2:2, 3:3, 4:4}, - 2:5, - 3:6, - 4:7} - -KEEDOX_LAYOUT = {1:1, - 2:2, - 3:3, - 4:{1:4, 2:5, 3:6, 4:7}} - -VIA_LAYOUT = {1:1, - 2:2, - 3:3, - 4:{1:4, 2:5, 3:6, 4:7}} +PLUGABLE_7PORT_LAYOUT = {1: 7, 2: 6, 3: 5, 4: {1: 4, 2: 3, 3: 2, 4: 1}} + +PLUGABLE_7PORT_USB3_LAYOUT = {1: {1: 1, 2: 2, 3: 3, 4: 4}, 2: 5, 3: 6, 4: 7} + +KEEDOX_LAYOUT = {1: 1, 2: 2, 3: 3, 4: {1: 4, 2: 5, 3: 6, 4: 7}} + +VIA_LAYOUT = {1: 1, 2: 2, 3: 3, 4: {1: 4, 2: 5, 3: 6, 4: 7}} + class HubType(object): def __init__(self, id_func, port_mapping): @@ -92,6 +81,7 @@ class HubType(object): else: yield (physical, node.PortToDevice(virtual)) + def _is_plugable_7port_hub(node): """Check if a node is a Plugable 7-Port Hub (Model USB2-HUB7BC) @@ -104,11 +94,13 @@ def _is_plugable_7port_hub(node): return False return '1a40:0101' in node.PortToDevice(4).desc + # Plugable 7-Port USB-3 Hubs show up twice in the USB devices list; they have # two different "branches", one which has USB2 devices and one which has # USB3 devices. The "part2" is the "USB-2" branch of the hub, the # "part3" is the "USB-3" branch of the hub. + def _is_plugable_7port_usb3_part2_hub(node): """Check if a node is the "USB2 branch" of a Plugable 7-Port USB-3 Hub (Model USB3-HUB7BC) @@ -121,6 +113,7 @@ def _is_plugable_7port_usb3_part2_hub(node): return False return '2109:2811' in node.PortToDevice(1).desc + def _is_plugable_7port_usb3_part3_hub(node): """Check if a node is the "USB3 branch" of a Plugable 7-Port USB-3 Hub (Model USB3-HUB7BC) @@ -133,6 +126,7 @@ def _is_plugable_7port_usb3_part3_hub(node): return False return '2109:8110' in node.PortToDevice(1).desc + def _is_keedox_hub(node): """Check if a node is a Keedox hub. The topology of this device is a 4-port hub, @@ -144,6 +138,7 @@ def _is_keedox_hub(node): return False return '0bda:5411' in node.PortToDevice(4).desc + def _is_via_hub(node): """Check if a node is a Via Labs hub. The topology of this device is a 4-port hub, @@ -153,8 +148,8 @@ def _is_via_hub(node): return False if not node.HasPort(4): return False - return ('2109:2812' in node.PortToDevice(4).desc or - '2109:0812' in node.PortToDevice(4).desc) + return ('2109:2812' in node.PortToDevice(4).desc + or '2109:0812' in node.PortToDevice(4).desc) PLUGABLE_7PORT = HubType(_is_plugable_7port_hub, PLUGABLE_7PORT_LAYOUT) @@ -165,11 +160,11 @@ PLUGABLE_7PORT_USB3_PART3 = HubType(_is_plugable_7port_usb3_part3_hub, KEEDOX = HubType(_is_keedox_hub, KEEDOX_LAYOUT) VIA = HubType(_is_via_hub, VIA_LAYOUT) -ALL_HUBS = [PLUGABLE_7PORT, - PLUGABLE_7PORT_USB3_PART2, - PLUGABLE_7PORT_USB3_PART3, - KEEDOX, - VIA] +ALL_HUBS = [ + PLUGABLE_7PORT, PLUGABLE_7PORT_USB3_PART2, PLUGABLE_7PORT_USB3_PART3, + KEEDOX, VIA +] + def GetHubType(type_name): if type_name == 'plugable_7port': diff --git a/catapult/devil/devil/utils/watchdog_timer.py b/catapult/devil/devil/utils/watchdog_timer.py index bff1f8cc..2c8a2250 100644 --- a/catapult/devil/devil/utils/watchdog_timer.py +++ b/catapult/devil/devil/utils/watchdog_timer.py @@ -1,7 +1,6 @@ # Copyright 2013 The Chromium Authors. All rights reserved. # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - """WatchdogTimer timeout objects.""" import time diff --git a/catapult/devil/devil/utils/zip_utils.py b/catapult/devil/devil/utils/zip_utils.py index e1f812b7..35c59056 100644 --- a/catapult/devil/devil/utils/zip_utils.py +++ b/catapult/devil/devil/utils/zip_utils.py @@ -9,18 +9,19 @@ import os import sys import zipfile +_DEVIL_ROOT_DIR = os.path.abspath( + os.path.join(os.path.dirname(__file__), '..', '..')) if __name__ == '__main__': - _DEVIL_ROOT_DIR = os.path.abspath( - os.path.join(os.path.dirname(__file__), '..', '..')) - _PY_UTILS_ROOT_DIR = os.path.abspath( - os.path.join(_DEVIL_ROOT_DIR, '..', 'common', 'py_utils')) - sys.path.extend((_DEVIL_ROOT_DIR, _PY_UTILS_ROOT_DIR)) + sys.path.append(_DEVIL_ROOT_DIR) + +_PY_UTILS_ROOT_DIR = os.path.abspath( + os.path.join(_DEVIL_ROOT_DIR, '..', 'common', 'py_utils')) +sys.path.append(_PY_UTILS_ROOT_DIR) from devil import base_error from devil.utils import cmd_helper from py_utils import tempfile_ext - logger = logging.getLogger(__name__) @@ -72,16 +73,16 @@ def WriteZipFile(zip_path, zip_contents): ZipFailedError on failure. """ zip_spec = { - 'zip_path': zip_path, - 'zip_contents': zip_contents, + 'zip_path': zip_path, + 'zip_contents': zip_contents, } with tempfile_ext.NamedTemporaryDirectory() as tmpdir: json_path = os.path.join(tmpdir, 'zip_spec.json') with open(json_path, 'w') as json_file: json.dump(zip_spec, json_file) - ret, output, error = cmd_helper.GetCmdStatusOutputAndError([ - sys.executable, os.path.abspath(__file__), - '--zip-spec', json_path]) + ret, output, error = cmd_helper.GetCmdStatusOutputAndError( + [sys.executable, + os.path.abspath(__file__), '--zip-spec', json_path]) if ret != 0: exc_msg = ['Failed to create %s' % zip_path] diff --git a/catapult/devil/devil/utils/zip_utils_test.py b/catapult/devil/devil/utils/zip_utils_test.py index 4564e3f1..96a55f55 100644 --- a/catapult/devil/devil/utils/zip_utils_test.py +++ b/catapult/devil/devil/utils/zip_utils_test.py @@ -11,7 +11,6 @@ from py_utils import tempfile_ext class WriteZipFileTest(unittest.TestCase): - def testSimple(self): with tempfile_ext.NamedTemporaryDirectory() as working_dir: file1 = os.path.join(working_dir, 'file1.txt') @@ -23,8 +22,8 @@ class WriteZipFileTest(unittest.TestCase): f2.write('file2') zip_tuples = [ - (file1, 'foo/file1.txt'), - (file2, 'bar/file2.txt'), + (file1, 'foo/file1.txt'), + (file2, 'bar/file2.txt'), ] zip_path = os.path.join(working_dir, 'out.zip') @@ -34,10 +33,8 @@ class WriteZipFileTest(unittest.TestCase): actual = zipfile.ZipFile(zip_path) expected_files = [ - 'foo/file1.txt', - 'bar/file2.txt', + 'foo/file1.txt', + 'bar/file2.txt', ] - self.assertEquals( - sorted(expected_files), - sorted(actual.namelist())) + self.assertEquals(sorted(expected_files), sorted(actual.namelist())) diff --git a/catapult/devil/docs/adb_wrapper.md b/catapult/devil/docs/adb_wrapper.md index a8dc3b05..3ca8aa68 100644 --- a/catapult/devil/docs/adb_wrapper.md +++ b/catapult/devil/docs/adb_wrapper.md @@ -1,132 +1,58 @@ # [devil.android.sdk.adb_wrapper](https://github.com/catapult-project/catapult/blob/master/devil/devil/android/sdk/adb_wrapper.py) -*This page was autogenerated by `devil/utils/markdown.py --module-link https://github.com/catapult-project/catapult/blob/master/devil/devil/android/sdk/adb_wrapper.py`* +*This page was autogenerated. Run `devil/bin/generate_md_docs` to update* -## DeviceStat - -DeviceStat(st\_mode, st\_size, st\_time) -### DeviceStat.\_\_repr\_\_ - -Return a nicely formatted representation string -### DeviceStat.\_\_getnewargs\_\_ - -Return self as a plain tuple. Used by copy and pickle. -### DeviceStat.\_\_getstate\_\_ - -Exclude the OrderedDict from pickling ## AdbWrapper A wrapper around a local Android Debug Bridge executable. -### AdbWrapper.GetDeviceSerial - -Gets the device serial number associated with this object. -``` - Returns: - Device serial number as a string. -``` - - -### AdbWrapper.Push - -Pushes a file from the host to the device. -``` - Args: - local: Path on the host filesystem. - remote: Path on the device filesystem. - timeout: (optional) Timeout per try in seconds. - retries: (optional) Number of retries to attempt. -``` - - -### AdbWrapper.Pull +### AdbWrapper.Backup -Pulls a file from the device to the host. +Write an archive of the device's data to |path|. ``` Args: - remote: Path on the device filesystem. - local: Path on the host filesystem. + path: Local path to store the backup file. + packages: List of to packages to be backed up. + apk: (optional) If set include the .apk files in the archive. + shared: (optional) If set buckup the device's SD card. + nosystem: (optional) If set exclude system applications. + include_all: (optional) If set back up all installed applications and + |packages| is optional. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. ``` -### AdbWrapper.Shell +### AdbWrapper.DisableVerity -Runs a shell command on the device. +Disable Marshmallow's Verity security feature. ``` - Args: - command: A string with the shell command to run. - expect_status: (optional) Check that the command's exit status matches - this value. Default is 0. If set to None the test is skipped. - timeout: (optional) Timeout per try in seconds. - retries: (optional) Number of retries to attempt. - Returns: - The output of the shell command as a string. - - Raises: - device_errors.AdbCommandFailedError: If the exit status doesn't match - |expect_status|. + The output of the disable-verity command as a string. ``` -### AdbWrapper.IterShell - -Runs a shell command and returns an iterator over its output lines. -``` - Args: - command: A string with the shell command to run. - timeout: Timeout in seconds. +### AdbWrapper.Emu - Yields: - The output of the command line by line. +Runs an emulator console command. ``` + See http://developer.android.com/tools/devices/emulator.html#console - -### AdbWrapper.Ls - -List the contents of a directory on the device. -``` Args: - path: Path on the device filesystem. + cmd: The command to run on the emulator console. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. Returns: - A list of pairs (filename, stat) for each file found in the directory, - where the stat object has the properties: st_mode, st_size, and st_time. - - Raises: - AdbCommandFailedError if |path| does not specify a valid and accessible - directory in the device, or the output of "adb ls" command is less - than four columns + The output of the emulator console command. ``` -### AdbWrapper.Logcat +### AdbWrapper.EnableVerity -Get an iterable over the logcat output. +Enable Marshmallow's Verity security feature. ``` - Args: - clear: If true, clear the logcat. - dump: If true, dump the current logcat contents. - filter_specs: If set, a list of specs to filter the logcat. - logcat_format: If set, the format in which the logcat should be output. - Options include "brief", "process", "tag", "thread", "raw", "time", - "threadtime", and "long" - ring_buffer: If set, a list of alternate ring buffers to request. - Options include "main", "system", "radio", "events", "crash" or "all". - The default is equivalent to ["main", "system", "crash"]. - iter_timeout: If set and neither clear nor dump is set, the number of - seconds to wait between iterations. If no line is found before the - given number of seconds elapses, the iterable will yield None. - timeout: (optional) If set, timeout per try in seconds. If clear or dump - is set, defaults to DEFAULT_TIMEOUT. - retries: (optional) If clear or dump is set, the number of retries to - attempt. Otherwise, does nothing. - - Yields: - logcat output line by line. + Returns: + The output of the enable-verity command as a string. ``` @@ -153,6 +79,18 @@ Forward socket connections from the local socket to the remote socket. ``` +### AdbWrapper.ForwardList + +List all currently forwarded socket connections. +``` + Args: + timeout: (optional) Timeout per try in seconds. + retries: (optional) Number of retries to attempt. + Returns: + The output of adb forward --list as a string. +``` + + ### AdbWrapper.ForwardRemove Remove a forward socket connection. @@ -164,28 +102,38 @@ Remove a forward socket connection. ``` -### AdbWrapper.ForwardList +### AdbWrapper.GetDevPath -List all currently forwarded socket connections. +Gets the device path. ``` Args: timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. + Returns: - The output of adb forward --list as a string. + The device path (e.g. usb:3-4) ``` -### AdbWrapper.JDWP +### AdbWrapper.GetDeviceSerial -List of PIDs of processes hosting a JDWP transport. +Gets the device serial number associated with this object. +``` + Returns: + Device serial number as a string. +``` + + +### AdbWrapper.GetState + +Get device state. ``` Args: timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. Returns: - A list of PIDs as strings. + One of 'offline', 'bootloader', or 'device'. ``` @@ -199,6 +147,10 @@ Install an apk on the device. allow_downgrade: (optional) If set, allows for downgrades. reinstall: (optional) If set reinstalls the app, keeping its data. sd_card: (optional) If set installs on the SD card. + streaming: (optional) If not set, use default way to install. + If True, performs streaming install. + If False, app is pushed to device and be installed from there. + Note this option is not supported prior to adb version 1.0.40 timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. ``` @@ -215,96 +167,138 @@ Install an apk with splits on the device. sd_card: (optional) If set installs on the SD card. allow_downgrade: (optional) Allow versionCode downgrade. partial: (optional) Package ID if apk_paths doesn't include all .apks. + streaming: (optional) If not set, use default way to install. + If True, performs streaming install. + If False, app is pushed to device and be installed from there. + Note this option is not supported prior to adb version 1.0.40 timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. ``` -### AdbWrapper.Uninstall +### AdbWrapper.IterShell -Remove the app |package| from the device. +Runs a shell command and returns an iterator over its output lines. ``` Args: - package: The package to uninstall. - keep_data: (optional) If set keep the data and cache directories. - timeout: (optional) Timeout per try in seconds. - retries: (optional) Number of retries to attempt. + command: A string with the shell command to run. + timeout: Timeout in seconds. + + Yields: + The output of the command line by line. ``` -### AdbWrapper.Backup +### AdbWrapper.JDWP -Write an archive of the device's data to |path|. +List of PIDs of processes hosting a JDWP transport. ``` Args: - path: Local path to store the backup file. - packages: List of to packages to be backed up. - apk: (optional) If set include the .apk files in the archive. - shared: (optional) If set buckup the device's SD card. - nosystem: (optional) If set exclude system applications. - include_all: (optional) If set back up all installed applications and - |packages| is optional. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. + + Returns: + A list of PIDs as strings. ``` -### AdbWrapper.Restore +### AdbWrapper.Logcat -Restore device contents from the backup archive. +Get an iterable over the logcat output. ``` Args: - path: Host path to the backup archive. + clear: If true, clear the logcat. + dump: If true, dump the current logcat contents. + filter_specs: If set, a list of specs to filter the logcat. + logcat_format: If set, the format in which the logcat should be output. + Options include "brief", "process", "tag", "thread", "raw", "time", + "threadtime", and "long" + ring_buffer: If set, a list of alternate ring buffers to request. + Options include "main", "system", "radio", "events", "crash" or "all". + The default is equivalent to ["main", "system", "crash"]. + iter_timeout: If set and neither clear nor dump is set, the number of + seconds to wait between iterations. If no line is found before the + given number of seconds elapses, the iterable will yield None. + check_error: Whether to check the exit status of the logcat command. + timeout: (optional) If set, timeout per try in seconds. If clear or dump + is set, defaults to DEFAULT_TIMEOUT. + retries: (optional) If clear or dump is set, the number of retries to + attempt. Otherwise, does nothing. + + Yields: + logcat output line by line. +``` + + +### AdbWrapper.Ls + +List the contents of a directory on the device. +``` + Args: + path: Path on the device filesystem. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. + + Returns: + A list of pairs (filename, stat) for each file found in the directory, + where the stat object has the properties: st_mode, st_size, and st_time. + + Raises: + AdbCommandFailedError if |path| does not specify a valid and accessible + directory in the device, or the output of "adb ls" command is less + than four columns ``` -### AdbWrapper.WaitForDevice +### AdbWrapper.Pull -Block until the device is online. +Pulls a file from the device to the host. ``` Args: + remote: Path on the device filesystem. + local: Path on the host filesystem. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. ``` -### AdbWrapper.GetState +### AdbWrapper.Push -Get device state. +Pushes a file from the host to the device. ``` Args: + local: Path on the host filesystem. + remote: Path on the device filesystem. + sync: (optional) Whether to only push files that are newer on the host. + Not supported when using adb prior to 1.0.39. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. - Returns: - One of 'offline', 'bootloader', or 'device'. + Raises: + AdbVersionError if sync=True with versions of adb prior to 1.0.39. ``` -### AdbWrapper.GetDevPath +### AdbWrapper.Reboot -Gets the device path. +Reboots the device. ``` Args: + to_bootloader: (optional) If set reboots to the bootloader. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. - - Returns: - The device path (e.g. usb:3-4) ``` ### AdbWrapper.Remount Remounts the /system partition on the device read-write. -### AdbWrapper.Reboot +### AdbWrapper.Restore -Reboots the device. +Restore device contents from the backup archive. ``` Args: - to_bootloader: (optional) If set reboots to the bootloader. + path: Host path to the backup archive. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. ``` @@ -320,34 +314,58 @@ Restarts the adbd daemon with root permissions, if possible. ``` -### AdbWrapper.Emu +### AdbWrapper.Shell -Runs an emulator console command. +Runs a shell command on the device. ``` - See http://developer.android.com/tools/devices/emulator.html#console - Args: - cmd: The command to run on the emulator console. + command: A string with the shell command to run. + expect_status: (optional) Check that the command's exit status matches + this value. Default is 0. If set to None the test is skipped. timeout: (optional) Timeout per try in seconds. retries: (optional) Number of retries to attempt. Returns: - The output of the emulator console command. + The output of the shell command as a string. + + Raises: + device_errors.AdbCommandFailedError: If the exit status doesn't match + |expect_status|. ``` -### AdbWrapper.DisableVerity +### AdbWrapper.StartShell -Disable Marshmallow's Verity security feature -### AdbWrapper.EnableVerity +Starts a subprocess on the device and returns a handle to the process. +``` + Args: + args: A sequence of program arguments. The executable to run is the first + item in the sequence. -Enable Marshmallow's Verity security feature -### AdbWrapper.\_\_init\_\_ + Returns: + An instance of subprocess.Popen associated with the live process. +``` -Initializes the AdbWrapper. + +### AdbWrapper.Uninstall + +Remove the app |package| from the device. ``` Args: - device_serial: The device serial number as a string. + package: The package to uninstall. + keep_data: (optional) If set keep the data and cache directories. + timeout: (optional) Timeout per try in seconds. + retries: (optional) Number of retries to attempt. +``` + + +### AdbWrapper.WaitForDevice + +Block until the device is online. +``` + Args: + timeout: (optional) Timeout per try in seconds. + retries: (optional) Number of retries to attempt. ``` @@ -363,6 +381,17 @@ Consider instances equal if they refer to the same device. ``` +### AdbWrapper.\_\_init\_\_ + +Initializes the AdbWrapper. +``` + Args: + device_serial: The device serial number as a string. +``` + + +### AdbWrapper.\_\_repr\_\_ + ### AdbWrapper.\_\_str\_\_ The string representation of an instance. @@ -372,8 +401,18 @@ The string representation of an instance. ``` -### AdbWrapper.\_\_repr\_\_ +## DeviceStat +DeviceStat(st\_mode, st\_size, st\_time) +### DeviceStat.\_\_getnewargs\_\_ + +Return self as a plain tuple. Used by copy and pickle. +### DeviceStat.\_\_getstate\_\_ + +Exclude the OrderedDict from pickling +### DeviceStat.\_\_repr\_\_ + +Return a nicely formatted representation string ### VerifyLocalFileExists Verifies a local file exists. diff --git a/catapult/devil/docs/device_blacklist.md b/catapult/devil/docs/device_denylist.md index c6eed514..0d654cdc 100644 --- a/catapult/devil/docs/device_blacklist.md +++ b/catapult/devil/docs/device_denylist.md @@ -3,11 +3,11 @@ found in the LICENSE file. --> -# Devil: Device Blacklist +# Devil: Device Denylist ## What is it? -The device blacklist is a per-run list of devices detected to be in a known bad +The device denylist is a per-run list of devices detected to be in a known bad state along with the reason they are suspected of being in a bad state (offline, not responding, etc). It is stored as a json file. This gets reset every run during the device recovery step (currently part of `bb_device_status_check`). @@ -15,13 +15,13 @@ during the device recovery step (currently part of `bb_device_status_check`). ## Bots On bots, this is normally found at `//out/bad_devices.json`. If you are having -problems with blacklisted devices locally even though a device is in a good +problems with denylisted devices locally even though a device is in a good state, you can safely delete this file. -# Tools for interacting with device black list. +# Tools for interacting with device deny list. -You can interact with the device blacklist via [devil.android.device\_blacklist](https://cs.chromium.org/chromium/src/third_party/catapult/devil/devil/android/device_blacklist.py). -This allows for any interaction you would need with a device blacklist: +You can interact with the device denylist via [devil.android.device\_denylist](https://cs.chromium.org/chromium/src/third_party/catapult/devil/devil/android/device_denylist.py). +This allows for any interaction you would need with a device denylist: - Reading - Writing @@ -30,28 +30,28 @@ This allows for any interaction you would need with a device blacklist: An example usecase of this is: ```python -from devil.android import device_blacklist - -blacklist = device_blacklist.Blacklist(blacklist_path) -blacklisted_devices = blacklist.Read() -for device in blacklisted_devices: - print 'Device %s is blacklisted' % device -blacklist.Reset() -new_blacklist = {'device_id1': {'timestamp': ts, 'reason': reason}} -blacklist.Write(new_blacklist) -blacklist.Extend([device_2, device_3], reason='Reason for blacklisting') +from devil.android import device_denylist + +denylist = device_denylist.Denylist(denylist_path) +denylisted_devices = denylist.Read() +for device in denylisted_devices: + print 'Device %s is denylisted' % device +denylist.Reset() +new_denylist = {'device_id1': {'timestamp': ts, 'reason': reason}} +denylist.Write(new_denylist) +denylist.Extend([device_2, device_3], reason='Reason for denylisting') ``` ## Where it is used. -The blacklist file path is passed directly to the following scripts in chromium: +The denylist file path is passed directly to the following scripts in chromium: - [test\_runner.py](https://cs.chromium.org/chromium/src/build/android/test_runner.py) - [provision\_devices.py](https://cs.chromium.org/chromium/src/build/android/provision_devices.py) - [bb\_device\_status\_check.py](https://cs.chromium.org/chromium/src/build/android/buildbot/bb_device_status_check.py) -The blacklist is also used in the following scripts: +The denylist is also used in the following scripts: - [device\_status.py](https://cs.chromium.org/chromium/src/third_party/catapult/devil/devil/android/tools/device_status.py) - [device\_recovery.py](https://cs.chromium.org/chromium/src/third_party/catapult/devil/devil/android/tools/device_recovery.py) diff --git a/catapult/devil/docs/device_utils.md b/catapult/devil/docs/device_utils.md index a6e89a70..960ca894 100644 --- a/catapult/devil/docs/device_utils.md +++ b/catapult/devil/docs/device_utils.md @@ -1,133 +1,297 @@ # [devil.android.device_utils](https://github.com/catapult-project/catapult/blob/master/devil/devil/android/device_utils.py) -*This page was autogenerated by `devil/utils/markdown.py --module-link https://github.com/catapult-project/catapult/blob/master/devil/devil/android/device_utils.py`* +*This page was autogenerated. Run `devil/bin/generate_md_docs` to update* ## DeviceUtils -### DeviceUtils.\_\_init\_\_ +### DeviceUtils.BroadcastIntent -DeviceUtils constructor. +Send a broadcast intent. ``` Args: - device: Either a device serial, an existing AdbWrapper instance, or an - an existing AndroidCommands instance. - enable_device_files_cache: For PushChangedFiles(), cache checksums of - pushed files rather than recomputing them on a subsequent call. - default_timeout: An integer containing the default number of seconds to - wait for an operation to complete if no explicit value is provided. - default_retries: An integer containing the default number or times an - operation should be retried on failure if no explicit value is provided. + intent: An Intent to broadcast. + timeout: timeout in seconds + retries: number of retries + + Raises: + CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. ``` -### DeviceUtils.\_\_eq\_\_ +### DeviceUtils.ChangeOwner -Checks whether |other| refers to the same device as |self|. +Changes file system ownership for permissions. ``` Args: - other: The object to compare to. This can be a basestring, an instance - of adb_wrapper.AdbWrapper, or an instance of DeviceUtils. - Returns: - Whether |other| refers to the same device as |self|. + owner_group: New owner and group to assign. Note that this should be a + string in the form user[.group] where the group is option. + paths: Paths to change ownership of. + + Note that the -R recursive option is not supported by all Android + versions. ``` -### DeviceUtils.\_\_lt\_\_ +### DeviceUtils.ChangeSecurityContext -Compares two instances of DeviceUtils. +Changes the SELinux security context for files. ``` - This merely compares their serial numbers. + Args: + security_context: The new security context as a string + paths: Paths to change the security context of. + + Note that the -R recursive option is not supported by all Android + versions. +``` + +### DeviceUtils.ClearApplicationState + +Clear all state for the given package. +``` Args: - other: The instance of DeviceUtils to compare to. + package: A string containing the name of the package to stop. + permissions: List of permissions to set after clearing data. + timeout: timeout in seconds + retries: number of retries + + Raises: + CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. +``` + + +### DeviceUtils.ClearCache + +Clears all caches. +### DeviceUtils.DismissCrashDialogIfNeeded + +Dismiss the error/ANR dialog if present. +``` + Returns: Name of the crashed package if a dialog is focused, + None otherwise. +``` + + +### DeviceUtils.DumpCacheData + +Dumps the current cache state to a string. +``` + Args: + timeout: timeout in seconds + retries: number of retries + Returns: - Whether |self| is less than |other|. + A serialized cache as a string. ``` -### DeviceUtils.\_\_str\_\_ +### DeviceUtils.EnableRoot -Returns the device serial. -### DeviceUtils.NeedsSU +Restarts adbd with root privileges. +``` + Args: + timeout: timeout in seconds + retries: number of retries -Checks whether 'su' is needed to access protected resources. + Raises: + CommandFailedError if root could not be enabled. + CommandTimeoutError on timeout. +``` + + +### DeviceUtils.FileExists + +Checks whether the given file exists on the device. +``` + Arguments are the same as PathExists. +``` + + +### DeviceUtils.FileSize + +Get the size of a file on the device. ``` + Note: This is implemented by parsing the output of the 'ls' command on + the device. On some Android versions, when passing a directory or special + file, the size is *not* reported and this function will throw an exception. + Args: + device_path: A string containing the path of a file on the device. + as_root: A boolean indicating whether the to use root privileges to + access the file information. timeout: timeout in seconds retries: number of retries Returns: - True if 'su' is available on the device and is needed to to access - protected resources; False otherwise if either 'su' is not available - (e.g. because the device has a user build), or not needed (because adbd - already has root privileges). + The size of the file in bytes. Raises: + CommandFailedError if device_path cannot be found on the device, or + its size cannot be determited for some reason. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.IsOnline +### DeviceUtils.ForceStop -Checks whether the device is online. +Close the application. +``` + Args: + package: A string containing the name of the package to stop. + timeout: timeout in seconds + retries: number of retries + + Raises: + CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. +``` + + +### DeviceUtils.GetABI + +Gets the device main ABI. ``` Args: timeout: timeout in seconds retries: number of retries Returns: - True if the device is online, False otherwise. + The device's main ABI name. For supported ABIs, the return value will be + one of the values defined in devil.android.ndk.abis. Raises: CommandTimeoutError on timeout. ``` -### DeviceUtils.HasRoot +### DeviceUtils.GetAppWritablePath -Checks whether or not adbd has root privileges. +Get a path that on the device's SD card that apps can write. ``` Args: timeout: timeout in seconds retries: number of retries Returns: - True if adbd has root privileges, False otherwise. + A app-writeable path on the device's SD card. Raises: + CommandFailedError if the external storage path could not be determined. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.EnableRoot +### DeviceUtils.GetApplicationDataDirectory -Restarts adbd with root privileges. +Get the data directory on the device for the given package. +``` + Args: + package: Name of the package. + + Returns: + The package's data directory. + Raises: + CommandFailedError if the package's data directory can't be found, + whether because it's not installed or otherwise. +``` + + +### DeviceUtils.GetApplicationPaths + +Get the paths of the installed apks on the device for the given package. +``` + Args: + package: Name of the package. + + Returns: + List of paths to the apks on the device for the given package. +``` + + +### DeviceUtils.GetApplicationPids + +Returns the PID or PIDs of a given process name. ``` + Note that the |process_name|, often the package name, must match exactly. + Args: + process_name: A string containing the process name to get the PIDs for. + at_most_one: A boolean indicating that at most one PID is expected to + be found. timeout: timeout in seconds retries: number of retries + Returns: + A list of the PIDs for the named process. If at_most_one=True returns + the single PID found or None otherwise. + Raises: - CommandFailedError if root could not be enabled. + CommandFailedError if at_most_one=True and more than one PID is found + for the named process. CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. ``` -### DeviceUtils.IsUserBuild +### DeviceUtils.GetApplicationTargetSdk -Checks whether or not the device is running a user build. +Get the targetSdkVersion of a package installed on the device. +``` + Args: + package: Name of the package. + + Returns: + A string with the targetSdkVersion or None if the package is not found on + the device. Note: this cannot always be cast to an integer. If this + application targets a pre-release SDK, this returns the version codename + instead (ex. "R"). +``` + + +### DeviceUtils.GetApplicationVersion + +Get the version name of a package installed on the device. +``` + Args: + package: Name of the package. + + Returns: + A string with the version name or None if the package is not found + on the device. +``` + + +### DeviceUtils.GetClientCache + +Returns client cache. +### DeviceUtils.GetCountry + +Returns the country setting on the device. +``` + DEPRECATED: Prefer GetLocale() instead. + + Args: + cache: Whether to use cached properties when available. +``` + + +### DeviceUtils.GetEnforce + +Get the current mode of SELinux. ``` Args: timeout: timeout in seconds retries: number of retries Returns: - True if the device is running a user build, False otherwise (i.e. if - it's running a userdebug build). + True (enforcing), False (permissive), or None (disabled). Raises: + CommandFailedError on failure. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` @@ -137,6 +301,9 @@ Checks whether or not the device is running a user build. Get the device's path to its SD card. ``` + Note: this path is read-only by apps in R+. Use GetAppWritablePath() to + obtain a path writable by apps. + Args: timeout: timeout in seconds retries: number of retries @@ -151,6 +318,9 @@ Get the device's path to its SD card. ``` +### DeviceUtils.GetFeatures + +Returns the features supported on the device. ### DeviceUtils.GetIMEI Get the device's IMEI. @@ -167,93 +337,190 @@ Get the device's IMEI. ``` -### DeviceUtils.GetApplicationPaths +### DeviceUtils.GetLanguage -Get the paths of the installed apks on the device for the given package. +Returns the language setting on the device. +``` + DEPRECATED: Prefer GetLocale() instead. + + Args: + cache: Whether to use cached properties when available. +``` + + +### DeviceUtils.GetLocale + +Returns the locale setting on the device. +``` + Args: + cache: Whether to use cached properties when available. + Returns: + A pair (language, country). +``` + + +### DeviceUtils.GetLogcatMonitor + +Returns a new LogcatMonitor associated with this device. +``` + Parameters passed to this function are passed directly to + |logcat_monitor.LogcatMonitor| and are documented there. +``` + + +### DeviceUtils.GetPackageArchitecture + +Get the architecture of a package installed on the device. ``` Args: package: Name of the package. Returns: - List of paths to the apks on the device for the given package. + A string with the architecture, or None if the package is missing. ``` -### DeviceUtils.TakeBugReport +### DeviceUtils.GetPids -Takes a bug report and dumps it to the specified path. +Returns the PIDs of processes containing the given name as substring. ``` - This doesn't use adb's bugreport option since its behavior is dependent on - both adb version and device OS version. To make it simpler, this directly - runs the bugreport command on the device itself and dumps the stdout to a - file. + DEPRECATED + + Note that the |process_name| is often the package name. Args: - path: Path on the host to drop the bug report. - timeout: (optional) Timeout per try in seconds. - retries: (optional) Number of retries to attempt. + process_name: A string containing the process name to get the PIDs for. + If missing returns PIDs for all processes. + timeout: timeout in seconds + retries: number of retries + + Returns: + A dict mapping process name to a list of PIDs for each process that + contained the provided |process_name|. + + Raises: + CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. ``` -### DeviceUtils.GetApplicationVersion +### DeviceUtils.GetProp -Get the version name of a package installed on the device. +Gets a property from the device. ``` Args: - package: Name of the package. + property_name: A string containing the name of the property to get from + the device. + cache: Whether to use cached properties when available. + timeout: timeout in seconds + retries: number of retries Returns: - A string with the version name or None if the package is not found - on the device. + The value of the device's |property_name| property. + + Raises: + CommandTimeoutError on timeout. ``` -### DeviceUtils.GetApplicationDataDirectory +### DeviceUtils.GetSecurityContextForPackage -Get the data directory on the device for the given package. +Gets the SELinux security context for the given package. ``` Args: package: Name of the package. + encrypted: Whether to check in the encrypted data directory + (/data/user_de/0/) or the unencrypted data directory (/data/data/). Returns: - The package's data directory. + The package's security context as a string, or None if not found. +``` + + +### DeviceUtils.GetTracingPath + +Gets tracing path from the device. +``` + Args: + timeout: timeout in seconds + retries: number of retries + + Returns: + /sys/kernel/debug/tracing for device with debugfs mount support; + /sys/kernel/tracing for device with tracefs support; + /sys/kernel/debug/tracing if support can't be determined. + Raises: - CommandFailedError if the package's data directory can't be found, - whether because it's not installed or otherwise. + CommandTimeoutError on timeout. ``` -### DeviceUtils.WaitUntilFullyBooted +### DeviceUtils.GetWebViewUpdateServiceDump -Wait for the device to fully boot. +Get the WebView update command sysdump on the device. ``` - This means waiting for the device to boot, the package manager to be - available, and the SD card to be ready. It can optionally mean waiting - for wifi to come up, too. + Returns: + A dictionary with these possible entries: + FallbackLogicEnabled: True|False + CurrentWebViewPackage: "package name" or None + MinimumWebViewVersionCode: int + WebViewPackages: Dict of installed WebView providers, mapping "package + name" to "reason it's valid/invalid." + + It may return an empty dictionary if device does not + support the "dumpsys webviewupdate" command. + + Raises: + CommandFailedError on failure. + CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. +``` + + +### DeviceUtils.GoHome + +Return to the home screen and obtain launcher focus. +``` + This command launches the home screen and attempts to obtain + launcher focus until the timeout is reached. Args: - wifi: A boolean indicating if we should wait for wifi to come up or not. timeout: timeout in seconds retries: number of retries Raises: - CommandFailedError on failure. - CommandTimeoutError if one of the component waits times out. - DeviceUnreachableError if the device becomes unresponsive. + CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. ``` -### DeviceUtils.Reboot +### DeviceUtils.GrantPermissions -Reboot the device. +### DeviceUtils.HasRoot + +Checks whether or not adbd has root privileges. ``` + A device is considered to have root if all commands are implicitly run + with elevated privileges, i.e. without having to use "su" to run them. + + Note that some devices do not allow this implicit privilige elevation, + but _can_ run commands as root just fine when done explicitly with "su". + To check if your device can run commands with elevated privileges at all + use: + + device.HasRoot() or device.NeedsSU() + + Luckily, for the most part you don't need to worry about this and using + RunShellCommand(cmd, as_root=True) will figure out for you the right + command incantation to run with elevated privileges. + Args: - block: A boolean indicating if we should wait for the reboot to complete. - wifi: A boolean indicating if we should wait for wifi to be enabled after - the reboot. The option has no effect unless |block| is also True. timeout: timeout in seconds retries: number of retries + Returns: + True if adbd has root privileges, False otherwise. + Raises: CommandTimeoutError on timeout. DeviceUnreachableError on missing device. @@ -262,18 +529,30 @@ Reboot the device. ### DeviceUtils.Install -Install an APK. +Install an APK or app bundle. ``` - Noop if an identical APK is already installed. + Noop if an identical APK is already installed. If installing a bundle, the + bundletools helper script (bin/*_bundle) should be used rather than the .aab + file. Args: - apk: An ApkHelper instance or string containing the path to the APK. + apk: An ApkHelper instance or string containing the path to the APK or + bundle. allow_downgrade: A boolean indicating if we should allow downgrades. reinstall: A boolean indicating if we should keep any existing app data. + Ignored if |apk| is a bundle. permissions: Set of permissions to set. If not set, finds permissions with apk helper. To set no permissions, pass []. timeout: timeout in seconds retries: number of retries + modules: An iterable containing specific bundle modules to install. + Error if set and |apk| points to an APK instead of a bundle. + fake_modules: An iterable containing specific bundle modules that should + have their apks copied to |MODULES_SRC_DIRECTORY_PATH| subdirectory + rather than installed. Thus the app can emulate SplitCompat while + running. This should not have any overlap with |modules|. + additional_locales: An iterable with additional locales to install for a + bundle. Raises: CommandFailedError if the installation fails. @@ -308,82 +587,69 @@ Install a split APK. ``` -### DeviceUtils.Uninstall +### DeviceUtils.IsApplicationInstalled -Remove the app |package\_name| from the device. +Determines whether a particular package is installed on the device. ``` - This is a no-op if the app is not already installed. + Args: + package: Name of the package. + + Returns: + True if the application is installed, False otherwise. +``` + +### DeviceUtils.IsOnline + +Checks whether the device is online. +``` Args: - package_name: The package to uninstall. - keep_data: (optional) Whether to keep the data and cache directories. - timeout: Timeout in seconds. - retries: Number of retries. + timeout: timeout in seconds + retries: number of retries + + Returns: + True if the device is online, False otherwise. Raises: - CommandFailedError if the uninstallation fails. - CommandTimeoutError if the uninstallation times out. - DeviceUnreachableError on missing device. + CommandTimeoutError on timeout. ``` -### DeviceUtils.RunShellCommand +### DeviceUtils.IsScreenOn -Run an ADB shell command. +Determines if screen is on. ``` - The command to run |cmd| should be a sequence of program arguments - (preferred) or a single string with a shell script to run. + Dumpsys input_method exposes screen on/off state. Below is an explination of + the states. - When |cmd| is a sequence, it is assumed to contain the name of the command - to run followed by its arguments. In this case, arguments are passed to the - command exactly as given, preventing any further processing by the shell. - This allows callers to easily pass arguments with spaces or special - characters without having to worry about quoting rules. Whenever possible, - it is recomended to pass |cmd| as a sequence. + Pre-L: + On: mScreenOn=true + Off: mScreenOn=false + L+: + On: mInteractive=true + Off: mInteractive=false - When |cmd| is passed as a single string, |shell| should be set to True. - The command will be interpreted and run by the shell on the device, - allowing the use of shell features such as pipes, wildcards, or variables. - Failing to set shell=True will issue a warning, but this will be changed - to a hard failure in the future (see: catapult:#3242). + Returns: + True if screen is on, false if it is off. + + Raises: + device_errors.CommandFailedError: If screen state cannot be found. +``` - This behaviour is consistent with that of command runners in cmd_helper as - well as Python's own subprocess.Popen. - TODO(perezju) Change the default of |check_return| to True when callers - have switched to the new behaviour. +### DeviceUtils.IsUserBuild +Checks whether or not the device is running a user build. +``` Args: - cmd: A sequence containing the command to run and its arguments, or a - string with a shell script to run (should also set shell=True). - shell: A boolean indicating whether shell features may be used in |cmd|. - check_return: A boolean indicating whether or not the return code should - be checked. - cwd: The device directory in which the command should be run. - env: The environment variables with which the command should be run. - run_as: A string containing the package as which the command should be - run. - as_root: A boolean indicating whether the shell command should be run - with root privileges. - single_line: A boolean indicating if only a single line of output is - expected. - large_output: Uses a work-around for large shell command output. Without - this large output will be truncated. - raw_output: Whether to only return the raw output - (no splitting into lines). timeout: timeout in seconds retries: number of retries Returns: - If single_line is False, the output of the command as a list of lines, - otherwise, a string with the unique line of output emmited by the command - (with the optional newline at the end stripped). + True if the device is running a user build, False otherwise (i.e. if + it's running a userdebug build). Raises: - AdbCommandFailedError if check_return is True and the exit code of - the command run on the device is non-zero. - CommandFailedError if single_line is True but the output contains two or - more lines. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` @@ -419,108 +685,127 @@ Kill all processes with the given name on the device. ``` -### DeviceUtils.StartActivity +### DeviceUtils.ListDirectory -Start package's activity on the device. +List all files on a device directory. ``` + Mirroring os.listdir (and most client expectations) the resulting list + does not include the special entries '.' and '..' even if they are present + in the directory. + Args: - intent_obj: An Intent object to send. - blocking: A boolean indicating whether we should wait for the activity to - finish launching. - trace_file_name: If present, a string that both indicates that we want to - profile the activity and contains the path to which the - trace should be saved. - force_stop: A boolean indicating whether we should stop the activity - before starting it. + device_path: A string containing the path of the directory on the device + to list. + as_root: A boolean indicating whether the to use root privileges to list + the directory contents. timeout: timeout in seconds retries: number of retries + Returns: + A list of filenames for all entries contained in the directory. + Raises: - CommandFailedError if the activity could not be started. + AdbCommandFailedError if |device_path| does not specify a valid and + accessible directory in the device. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.StartInstrumentation - -### DeviceUtils.BroadcastIntent +### DeviceUtils.ListProcesses -Send a broadcast intent. +Returns a list of tuples with info about processes on the device. ``` + This essentially parses the output of the |ps| command into convenient + ProcessInfo tuples. + Args: - intent: An Intent to broadcast. + process_name: A string used to filter the returned processes. If given, + only processes whose name have this value as a substring + will be returned. timeout: timeout in seconds retries: number of retries - Raises: - CommandTimeoutError on timeout. - DeviceUnreachableError on missing device. + Returns: + A list of ProcessInfo tuples with |name|, |pid|, and |ppid| fields. ``` -### DeviceUtils.GoHome +### DeviceUtils.LoadCacheData -Return to the home screen and obtain launcher focus. +Initializes the cache from data created using DumpCacheData. ``` - This command launches the home screen and attempts to obtain - launcher focus until the timeout is reached. + The cache is used only if its token matches the one found on the device. + This prevents a stale cache from being used (which can happen when sharing + devices). Args: + data: A previously serialized cache (string). timeout: timeout in seconds retries: number of retries - Raises: - CommandTimeoutError on timeout. - DeviceUnreachableError on missing device. + Returns: + Whether the cache was loaded. ``` -### DeviceUtils.ForceStop +### DeviceUtils.NeedsSU -Close the application. +Checks whether 'su' is needed to access protected resources. ``` Args: - package: A string containing the name of the package to stop. timeout: timeout in seconds retries: number of retries + Returns: + True if 'su' is available on the device and is needed to to access + protected resources; False otherwise if either 'su' is not available + (e.g. because the device has a user build), or not needed (because adbd + already has root privileges). + Raises: CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.ClearApplicationState +### DeviceUtils.PathExists -Clear all state for the given package. +Checks whether the given path(s) exists on the device. ``` Args: - package: A string containing the name of the package to stop. - permissions: List of permissions to set after clearing data. + device_path: A string containing the absolute path to the file on the + device, or an iterable of paths to check. + as_root: Whether root permissions should be use to check for the existence + of the given path(s). timeout: timeout in seconds retries: number of retries + Returns: + True if the all given paths exist on the device, False otherwise. + Raises: CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.SendKeyEvent +### DeviceUtils.PullFile -Sends a keycode to the device. +Pull a file from the device. ``` - See the devil.android.sdk.keyevent module for suitable keycode values. - Args: - keycode: A integer keycode to send to the device. + device_path: A string containing the absolute path of the file to pull + from the device. + host_path: A string containing the absolute path of the destination on + the host. + as_root: Whether root permissions should be used to pull the file. timeout: timeout in seconds retries: number of retries Raises: + CommandFailedError on failure. CommandTimeoutError on timeout. - DeviceUnreachableError on missing device. ``` @@ -538,79 +823,14 @@ Push files to the device, skipping files that don't need updating. |host_path| is an absolute path of a file or directory on the host that should be minimially pushed to the device, and |device_path| is an absolute path of the destination on the device. - timeout: timeout in seconds - retries: number of retries delete_device_stale: option to delete stale files on device - - Raises: - CommandFailedError on failure. - CommandTimeoutError on timeout. - DeviceUnreachableError on missing device. -``` - - -### DeviceUtils.FileExists - -Checks whether the given file exists on the device. -``` - Arguments are the same as PathExists. -``` - - -### DeviceUtils.PathExists - -Checks whether the given path(s) exists on the device. -``` - Args: - device_path: A string containing the absolute path to the file on the - device, or an iterable of paths to check. - as_root: Whether root permissions should be use to check for the existence - of the given path(s). - timeout: timeout in seconds - retries: number of retries - - Returns: - True if the all given paths exist on the device, False otherwise. - - Raises: - CommandTimeoutError on timeout. - DeviceUnreachableError on missing device. -``` - - -### DeviceUtils.RemovePath - -Removes the given path(s) from the device. -``` - Args: - device_path: A string containing the absolute path to the file on the - device, or an iterable of paths to check. - force: Whether to remove the path(s) with force (-f). - recursive: Whether to remove any directories in the path(s) recursively. - as_root: Whether root permissions should be use to remove the given - path(s). - rename: Whether to rename the path(s) before removing to help avoid - filesystem errors. See https://stackoverflow.com/questions/11539657 - timeout: timeout in seconds - retries: number of retries -``` - - -### DeviceUtils.PullFile - -Pull a file from the device. -``` - Args: - device_path: A string containing the absolute path of the file to pull - from the device. - host_path: A string containing the absolute path of the destination on - the host. timeout: timeout in seconds retries: number of retries Raises: CommandFailedError on failure. CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. ``` @@ -642,156 +862,143 @@ Reads the contents of a file from the device. ``` -### DeviceUtils.WriteFile +### DeviceUtils.Reboot -Writes |contents| to a file on the device. +Reboot the device. ``` Args: - device_path: A string containing the absolute path to the file to write - on the device. - contents: A string containing the data to write to the device. - as_root: A boolean indicating whether the write should be executed with - root privileges (if available). - force_push: A boolean indicating whether to force the operation to be - performed by pushing a file to the device. The default is, when the - contents are short, to pass the contents using a shell script instead. + block: A boolean indicating if we should wait for the reboot to complete. + wifi: A boolean indicating if we should wait for wifi to be enabled after + the reboot. + The option has no effect unless |block| is also True. + decrypt: A boolean indicating if we should wait for full-disk decryption + to complete after the reboot. + The option has no effect unless |block| is also True. timeout: timeout in seconds retries: number of retries Raises: - CommandFailedError if the file could not be written on the device. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.ListDirectory +### DeviceUtils.RemovePath -List all files on a device directory. +Removes the given path(s) from the device. ``` - Mirroring os.listdir (and most client expectations) the resulting list - does not include the special entries '.' and '..' even if they are present - in the directory. - Args: - device_path: A string containing the path of the directory on the device - to list. - as_root: A boolean indicating whether the to use root privileges to list - the directory contents. + device_path: A string containing the absolute path to the file on the + device, or an iterable of paths to check. + force: Whether to remove the path(s) with force (-f). + recursive: Whether to remove any directories in the path(s) recursively. + as_root: Whether root permissions should be use to remove the given + path(s). + rename: Whether to rename the path(s) before removing to help avoid + filesystem errors. See https://stackoverflow.com/questions/11539657 timeout: timeout in seconds retries: number of retries +``` - Returns: - A list of filenames for all entries contained in the directory. - Raises: - AdbCommandFailedError if |device_path| does not specify a valid and - accessible directory in the device. - CommandTimeoutError on timeout. - DeviceUnreachableError on missing device. +### DeviceUtils.RestartAdbd + +### DeviceUtils.RunShellCommand + +Run an ADB shell command. ``` + The command to run |cmd| should be a sequence of program arguments + (preferred) or a single string with a shell script to run. + When |cmd| is a sequence, it is assumed to contain the name of the command + to run followed by its arguments. In this case, arguments are passed to the + command exactly as given, preventing any further processing by the shell. + This allows callers to easily pass arguments with spaces or special + characters without having to worry about quoting rules. Whenever possible, + it is recomended to pass |cmd| as a sequence. -### DeviceUtils.StatDirectory + When |cmd| is passed as a single string, |shell| should be set to True. + The command will be interpreted and run by the shell on the device, + allowing the use of shell features such as pipes, wildcards, or variables. + Failing to set shell=True will issue a warning, but this will be changed + to a hard failure in the future (see: catapult:#3242). -List file and stat info for all entries on a device directory. -``` - Implementation notes: this is currently implemented by parsing the output - of 'ls -a -l' on the device. Whether possible and convenient, we attempt to - make parsing strict and return values mirroring those of the standard |os| - and |stat| Python modules. + This behaviour is consistent with that of command runners in cmd_helper as + well as Python's own subprocess.Popen. - Mirroring os.listdir (and most client expectations) the resulting list - does not include the special entries '.' and '..' even if they are present - in the directory. + TODO(crbug.com/1029769) Change the default of |check_return| to True when + callers have switched to the new behaviour. Args: - device_path: A string containing the path of the directory on the device - to list. - as_root: A boolean indicating whether the to use root privileges to list - the directory contents. + cmd: A sequence containing the command to run and its arguments, or a + string with a shell script to run (should also set shell=True). + shell: A boolean indicating whether shell features may be used in |cmd|. + check_return: A boolean indicating whether or not the return code should + be checked. + cwd: The device directory in which the command should be run. + env: The environment variables with which the command should be run. + run_as: A string containing the package as which the command should be + run. + as_root: A boolean indicating whether the shell command should be run + with root privileges. + single_line: A boolean indicating if only a single line of output is + expected. + large_output: Uses a work-around for large shell command output. Without + this large output will be truncated. + raw_output: Whether to only return the raw output + (no splitting into lines). timeout: timeout in seconds retries: number of retries Returns: - A list of dictionaries, each containing the following keys: - filename: A string with the file name. - st_mode: File permissions, use the stat module to interpret these. - st_nlink: Number of hard links (may be missing). - st_owner: A string with the user name of the owner. - st_group: A string with the group name of the owner. - st_rdev_pair: Device type as (major, minior) (only if inode device). - st_size: Size of file, in bytes (may be missing for non-regular files). - st_mtime: Time of most recent modification, in seconds since epoch - (although resolution is in minutes). - symbolic_link_to: If entry is a symbolic link, path where it points to; - missing otherwise. + If single_line is False, the output of the command as a list of lines, + otherwise, a string with the unique line of output emmited by the command + (with the optional newline at the end stripped). Raises: - AdbCommandFailedError if |device_path| does not specify a valid and - accessible directory in the device. + AdbCommandFailedError if check_return is True and the exit code of + the command run on the device is non-zero. + CommandFailedError if single_line is True but the output contains two or + more lines. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.StatPath +### DeviceUtils.SendKeyEvent -Get the stat attributes of a file or directory on the device. +Sends a keycode to the device. ``` + See the devil.android.sdk.keyevent module for suitable keycode values. + Args: - device_path: A string containing the path of a file or directory from - which to get attributes. - as_root: A boolean indicating whether the to use root privileges to - access the file information. + keycode: A integer keycode to send to the device. timeout: timeout in seconds retries: number of retries - Returns: - A dictionary with the stat info collected; see StatDirectory for details. - Raises: - CommandFailedError if device_path cannot be found on the device. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.FileSize +### DeviceUtils.SetEnforce -Get the size of a file on the device. +Modify the mode SELinux is running in. ``` - Note: This is implemented by parsing the output of the 'ls' command on - the device. On some Android versions, when passing a directory or special - file, the size is *not* reported and this function will throw an exception. - Args: - device_path: A string containing the path of a file on the device. - as_root: A boolean indicating whether the to use root privileges to - access the file information. + enabled: a boolean indicating whether to put SELinux in encorcing mode + (if True), or permissive mode (otherwise). timeout: timeout in seconds retries: number of retries - Returns: - The size of the file in bytes. - Raises: - CommandFailedError if device_path cannot be found on the device, or - its size cannot be determited for some reason. + CommandFailedError on failure. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.GetLanguage - -Returns the language setting on the device. -``` - Args: - cache: Whether to use cached properties when available. -``` - - ### DeviceUtils.SetJavaAsserts Enables or disables Java asserts. @@ -811,159 +1018,211 @@ Enables or disables Java asserts. ``` -### DeviceUtils.GetCountry +### DeviceUtils.SetProp -Returns the country setting on the device. +Sets a property on the device. ``` Args: - cache: Whether to use cached properties when available. + property_name: A string containing the name of the property to set on + the device. + value: A string containing the value to set to the property on the + device. + check: A boolean indicating whether to check that the property was + successfully set on the device. + timeout: timeout in seconds + retries: number of retries + + Raises: + CommandFailedError if check is true and the property was not correctly + set on the device (e.g. because it is not rooted). + CommandTimeoutError on timeout. ``` -### DeviceUtils.GetApplicationPids +### DeviceUtils.SetScreen -Returns the PID or PIDs of a given process name. +Turns screen on and off. ``` - Note that the |process_name|, often the package name, must match exactly. + Args: + on: bool to decide state to switch to. True = on False = off. +``` + + +### DeviceUtils.SetWebViewFallbackLogic + +Set whether WebViewUpdateService's "fallback logic" should be enabled. +``` + WebViewUpdateService has nonintuitive "fallback logic" for devices where + Monochrome (Chrome Stable) is preinstalled as the WebView provider, with a + "stub" (little-to-no code) implementation of standalone WebView. + + "Fallback logic" (enabled by default) is designed, in the case where the + user has disabled Chrome, to fall back to the stub standalone WebView by + enabling the package. The implementation plumbs through the Chrome APK until + Play Store installs an update with the full implementation. + + A surprising side-effect of "fallback logic" is that, immediately after + sideloading WebView, WebViewUpdateService re-disables the package and + uninstalls the update. This can prevent successfully using standalone + WebView for development, although "fallback logic" can be disabled on + userdebug/eng devices. + + Because this is only relevant for devices with the standalone WebView stub, + this command is only relevant on N-P (inclusive). + + You can determine if "fallback logic" is currently enabled by checking + FallbackLogicEnabled in the dictionary returned by + GetWebViewUpdateServiceDump. Args: - process_name: A string containing the process name to get the PIDs for. - at_most_one: A boolean indicating that at most one PID is expected to - be found. + enabled: bool - True for enabled, False for disabled timeout: timeout in seconds retries: number of retries - Returns: - A list of the PIDs for the named process. If at_most_one=True returns - the single PID found or None otherwise. - Raises: - CommandFailedError if at_most_one=True and more than one PID is found - for the named process. + CommandFailedError on failure. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.GetProp +### DeviceUtils.SetWebViewImplementation -Gets a property from the device. +Select the WebView implementation to the specified package. ``` Args: - property_name: A string containing the name of the property to get from - the device. - cache: Whether to use cached properties when available. + package_name: The package name of a WebView implementation. The package + must be already installed on the device. timeout: timeout in seconds retries: number of retries - Returns: - The value of the device's |property_name| property. - Raises: + CommandFailedError on failure. CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. ``` -### DeviceUtils.SetProp +### DeviceUtils.StartActivity -Sets a property on the device. +Start package's activity on the device. ``` Args: - property_name: A string containing the name of the property to set on - the device. - value: A string containing the value to set to the property on the - device. - check: A boolean indicating whether to check that the property was - successfully set on the device. + intent_obj: An Intent object to send. + blocking: A boolean indicating whether we should wait for the activity to + finish launching. + trace_file_name: If present, a string that both indicates that we want to + profile the activity and contains the path to which the + trace should be saved. + force_stop: A boolean indicating whether we should stop the activity + before starting it. timeout: timeout in seconds retries: number of retries Raises: - CommandFailedError if check is true and the property was not correctly - set on the device (e.g. because it is not rooted). + CommandFailedError if the activity could not be started. CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. ``` -### DeviceUtils.GetABI +### DeviceUtils.StartInstrumentation -Gets the device main ABI. +### DeviceUtils.StartService + +Start a service on the device. ``` Args: - timeout: timeout in seconds - retries: number of retries - - Returns: - The device's main ABI name. + intent_obj: An Intent object to send describing the service to start. + user_id: A specific user to start the service as, defaults to current. + timeout: Timeout in seconds. + retries: Number of retries Raises: + CommandFailedError if the service could not be started. CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. ``` -### DeviceUtils.GetPids +### DeviceUtils.StatDirectory -Returns the PIDs of processes containing the given name as substring. +List file and stat info for all entries on a device directory. ``` - Note that the |process_name| is often the package name. + Implementation notes: this is currently implemented by parsing the output + of 'ls -a -l' on the device. Whether possible and convenient, we attempt to + make parsing strict and return values mirroring those of the standard |os| + and |stat| Python modules. + + Mirroring os.listdir (and most client expectations) the resulting list + does not include the special entries '.' and '..' even if they are present + in the directory. Args: - process_name: A string containing the process name to get the PIDs for. - If missing returns PIDs for all processes. + device_path: A string containing the path of the directory on the device + to list. + as_root: A boolean indicating whether the to use root privileges to list + the directory contents. timeout: timeout in seconds retries: number of retries Returns: - A dict mapping process name to a list of PIDs for each process that - contained the provided |process_name|. + A list of dictionaries, each containing the following keys: + filename: A string with the file name. + st_mode: File permissions, use the stat module to interpret these. + st_nlink: Number of hard links (may be missing). + st_owner: A string with the user name of the owner. + st_group: A string with the group name of the owner. + st_rdev_pair: Device type as (major, minior) (only if inode device). + st_size: Size of file, in bytes (may be missing for non-regular files). + st_mtime: Time of most recent modification, in seconds since epoch + (although resolution is in minutes). + symbolic_link_to: If entry is a symbolic link, path where it points to; + missing otherwise. Raises: + AdbCommandFailedError if |device_path| does not specify a valid and + accessible directory in the device. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.GetLogcatMonitor - -Returns a new LogcatMonitor associated with this device. -``` - Parameters passed to this function are passed directly to - |logcat_monitor.LogcatMonitor| and are documented there. -``` - - -### DeviceUtils.GetEnforce +### DeviceUtils.StatPath -Get the current mode of SELinux. +Get the stat attributes of a file or directory on the device. ``` Args: + device_path: A string containing the path of a file or directory from + which to get attributes. + as_root: A boolean indicating whether the to use root privileges to + access the file information. timeout: timeout in seconds retries: number of retries Returns: - True (enforcing), False (permissive), or None (disabled). + A dictionary with the stat info collected; see StatDirectory for details. Raises: - CommandFailedError on failure. + CommandFailedError if device_path cannot be found on the device. CommandTimeoutError on timeout. DeviceUnreachableError on missing device. ``` -### DeviceUtils.SetEnforce +### DeviceUtils.TakeBugReport -Modify the mode SELinux is running in. +Takes a bug report and dumps it to the specified path. ``` - Args: - enabled: a boolean indicating whether to put SELinux in encorcing mode - (if True), or permissive mode (otherwise). - timeout: timeout in seconds - retries: number of retries + This doesn't use adb's bugreport option since its behavior is dependent on + both adb version and device OS version. To make it simpler, this directly + runs the bugreport command on the device itself and dumps the stdout to a + file. - Raises: - CommandFailedError on failure. - CommandTimeoutError on timeout. - DeviceUnreachableError on missing device. + Args: + path: Path on the host to drop the bug report. + timeout: (optional) Timeout per try in seconds. + retries: (optional) Number of retries to attempt. ``` @@ -988,84 +1247,128 @@ Takes a screenshot of the device. ``` -### DeviceUtils.DismissCrashDialogIfNeeded +### DeviceUtils.Uninstall -Dismiss the error/ANR dialog if present. -``` - Returns: Name of the crashed package if a dialog is focused, - None otherwise. +Remove the app |package\_name| from the device. ``` + This is a no-op if the app is not already installed. + Args: + package_name: The package to uninstall. + keep_data: (optional) Whether to keep the data and cache directories. + timeout: Timeout in seconds. + retries: Number of retries. -### DeviceUtils.GetClientCache + Raises: + CommandFailedError if the uninstallation fails. + CommandTimeoutError if the uninstallation times out. + DeviceUnreachableError on missing device. +``` -Returns client cache. -### DeviceUtils.LoadCacheData -Initializes the cache from data created using DumpCacheData. +### DeviceUtils.WaitUntilFullyBooted + +Wait for the device to fully boot. ``` - The cache is used only if its token matches the one found on the device. - This prevents a stale cache from being used (which can happen when sharing - devices). + This means waiting for the device to boot, the package manager to be + available, and the SD card to be ready. + It can optionally wait the following: + - Wait for wifi to come up. + - Wait for full-disk decryption to complete. Args: - data: A previously serialized cache (string). + wifi: A boolean indicating if we should wait for wifi to come up or not. + decrypt: A boolean indicating if we should wait for full-disk decryption + to complete. timeout: timeout in seconds retries: number of retries - Returns: - Whether the cache was loaded. + Raises: + CommandFailedError on failure. + CommandTimeoutError if one of the component waits times out. + DeviceUnreachableError if the device becomes unresponsive. ``` -### DeviceUtils.DumpCacheData +### DeviceUtils.WriteFile -Dumps the current cache state to a string. +Writes |contents| to a file on the device. ``` Args: + device_path: A string containing the absolute path to the file to write + on the device. + contents: A string containing the data to write to the device. + as_root: A boolean indicating whether the write should be executed with + root privileges (if available). + force_push: A boolean indicating whether to force the operation to be + performed by pushing a file to the device. The default is, when the + contents are short, to pass the contents using a shell script instead. timeout: timeout in seconds retries: number of retries - Returns: - A serialized cache as a string. + Raises: + CommandFailedError if the file could not be written on the device. + CommandTimeoutError on timeout. + DeviceUnreachableError on missing device. ``` -### DeviceUtils.RestartAdbd - -### DeviceUtils.GrantPermissions - -### DeviceUtils.IsScreenOn +### DeviceUtils.\_\_eq\_\_ -Determines if screen is on. +Checks whether |other| refers to the same device as |self|. +``` + Args: + other: The object to compare to. This can be a basestring, an instance + of adb_wrapper.AdbWrapper, or an instance of DeviceUtils. + Returns: + Whether |other| refers to the same device as |self|. ``` - Dumpsys input_method exposes screen on/off state. Below is an explination of - the states. - Pre-L: - On: mScreenOn=true - Off: mScreenOn=false - L+: - On: mInteractive=true - Off: mInteractive=false - Returns: - True if screen is on, false if it is off. +### DeviceUtils.\_\_init\_\_ - Raises: - device_errors.CommandFailedError: If screen state cannot be found. +DeviceUtils constructor. +``` + Args: + device: Either a device serial, an existing AdbWrapper instance, or an + an existing AndroidCommands instance. + enable_device_files_cache: For PushChangedFiles(), cache checksums of + pushed files rather than recomputing them on a subsequent call. + default_timeout: An integer containing the default number of seconds to + wait for an operation to complete if no explicit value is provided. + default_retries: An integer containing the default number or times an + operation should be retried on failure if no explicit value is provided. ``` -### DeviceUtils.SetScreen +### DeviceUtils.\_\_lt\_\_ -Turns screen on and off. +Compares two instances of DeviceUtils. ``` + This merely compares their serial numbers. + Args: - on: bool to decide state to switch to. True = on False = off. + other: The instance of DeviceUtils to compare to. + Returns: + Whether |self| is less than |other|. ``` +### DeviceUtils.\_\_str\_\_ + +Returns the device serial. +## ProcessInfo + +ProcessInfo(name, pid, ppid) +### ProcessInfo.\_\_getnewargs\_\_ + +Return self as a plain tuple. Used by copy and pickle. +### ProcessInfo.\_\_getstate\_\_ + +Exclude the OrderedDict from pickling +### ProcessInfo.\_\_repr\_\_ + +Return a nicely formatted representation string ### GetAVDs Returns a list of Android Virtual Devices. diff --git a/catapult/devil/docs/markdown.md b/catapult/devil/docs/markdown.md index 957dba7d..b2fec500 100644 --- a/catapult/devil/docs/markdown.md +++ b/catapult/devil/docs/markdown.md @@ -1,13 +1,13 @@ # [devil.utils.markdown](https://github.com/catapult-project/catapult/blob/master/devil/devil/utils/markdown.py) -*This page was autogenerated by `devil/utils/markdown.py --module-link https://github.com/catapult-project/catapult/blob/master/devil/devil/utils/markdown.py`* +*This page was autogenerated. Run `devil/bin/generate_md_docs` to update* ## MarkdownHelpAction -### MarkdownHelpAction.\_\_init\_\_ - ### MarkdownHelpAction.\_\_call\_\_ +### MarkdownHelpAction.\_\_init\_\_ + ## MarkdownHelpFormatter A really bare-bones argparse help formatter that generates valid markdown. @@ -31,27 +31,6 @@ A really bare-bones argparse help formatter that generates valid markdown. ### MarkdownHelpFormatter.start\_section -### md\_bold - -Returns markdown-formatted bold text. -### md\_code - -Returns a markdown-formatted code block in the given language. -### md\_escape - -Escapes \* and \_. -### md\_heading - -Returns markdown-formatted heading. -### md\_inline\_code - -Returns markdown-formatted inline code. -### md\_italic - -Returns markdown-formatted italic text. -### md\_link - -returns a markdown-formatted link. ### add\_md\_help\_argument Adds --md-help to the given argparse.ArgumentParser. @@ -77,20 +56,20 @@ Load a module given only the path name. ``` -### md\_module +### main -Write markdown documentation for a class. +Write markdown documentation for the module at the provided path. ``` - Documents public classes and functions. - Args: - class_obj: a types.TypeType object for the class that should be - documented. + raw_args: the raw command-line args. Usually sys.argv[1:]. Returns: - A list of markdown-formatted lines. + An integer exit code. 0 for success, non-zero for failure. ``` +### md\_bold + +Returns markdown-formatted bold text. ### md\_class Write markdown documentation for a class. @@ -105,6 +84,9 @@ Write markdown documentation for a class. ``` +### md\_code + +Returns a markdown-formatted code block in the given language. ### md\_docstring Write a markdown-formatted docstring. @@ -114,6 +96,9 @@ Write a markdown-formatted docstring. ``` +### md\_escape + +Escapes \* and \_. ### md\_function Write markdown documentation for a function. @@ -126,14 +111,28 @@ Write markdown documentation for a function. ``` -### main +### md\_heading -Write markdown documentation for the module at the provided path. +Returns markdown-formatted heading. +### md\_inline\_code + +Returns markdown-formatted inline code. +### md\_italic + +Returns markdown-formatted italic text. +### md\_link + +returns a markdown-formatted link. +### md\_module + +Write markdown documentation for a module. ``` + Documents public classes and functions. + Args: - raw_args: the raw command-line args. Usually sys.argv[1:]. + module_obj: a module object that should be documented. Returns: - An integer exit code. 0 for success, non-zero for failure. + A list of markdown-formatted lines. ``` |