Update to latest catapult (eae13a4)

$ git log --oneline --no-merges 3fe65c60..eae13a4b tracing systrace
7110f0868 rendering: Add pipeline:draw in the report.
3a995feda [systrace] Fix update_systrace_trace_viewer
2acd8e035 Removed an old restriction on timestamp ordering of trace events
134364159 [mali importer] Add Exynos Mali job event parser
5c2d9e757 renderer: Add breakdown metrics for compositor pipeline.
94294baed Reland "rendering: Add pipeline metrics."
600cb5aa8 tabs: Report tab-switching request delay.
e05f902a6 Revert "rendering: Add pipeline metrics."
b9813d2fd Attempt to fix flakiness in webview_startup perf tests
d8072e683 Fix lint errors for systrace
6a67dd77d Tracing: Fix native library stats for pre-N devices.
787a37892 rendering: Add pipeline metrics.
f07f60ef0 Drop Scheduler and FrameScheduler prefixes to save space.
0ddcb32d1 [TBMv2] Add POINT_ID reserved diagnostic
557d8dca5 [Systrace] Raise error when specified device is not found
be45355b4 Dashboard - Fix fast path in ReplaceSharedDiagnostic
66447ba64 trace-viewer: Show a link to codesearch.
68c3b1d86 Dashboard - Optimize histogram_set.ReplaceSharedDiagnostic
10041796a [ftrace importer] Use tid instead of pid for kernel threads
4c6f5a25f [ftrace importer] Fix recursive binder reply arguments
9e83c9e3c [TBMv2] Implement deduplication fast path for GenericSet
685c633d5 [TBMv2] Make timeInMsAutoFormat available in Python
0c92958e3 Refactor systrace/systrace/util.py to use device_utils
21acb2a82 Throw exception instead of dying in get_device_sdk_version
647cee134 Telemetry: frame times in TBMv2
0e9eb2f5c Android systrace: Fix printf/scanf format compiler warnings
bea867a76 Android systrace: Fix undeclared use of strerr
3766fc728 Android systrace: Fix undeclared references to POSIX APIs
d8e6a8203 [Telemetry] Remove GetRendererThreadFromTabId
7439a1c6b trace-recorder: Include 'viz' category for rendering.
3eb704bab Remove BattOr support from systrace
9f46241c6 Add documentation_link diagnostic
3836c556c androidStartupMetric: Allow multiple FCP events
f76f0b440 [Systrace] remove duplicate --target option.
7ddf0ed7f Add 90th percentile to results.html.
f989b6282 tracing: Add binder transaction size to events
fdc825f43 androidStartupMetric: support multiple browser starts
11513e359 Add convenient method for running trace processing code in python
87eefd4f1 [fuchsia] Add thread priority to fuchsia importer
32a31d2e9 Do not compute TTI without ResourceLoad
edf6c000f Add default bin boundaries for timeInMsAutoFormat.
c87e4baed Support multiple story tags in ChartJsonConverter.
09b82f4a5 Plumb story tags through Telemetry Values.
63f44ee35 tabs: Ignore the duration for the new metric.
6160b40d1 HistogramSet - Merge TagMap diagnostics in add_reserved_diagnostics
9861a71fa tabs: Switch back to async trace events.
4c1bea249 [ftrace importer] Add fence parser
a9d8c2947 tabs: Update the measurements.
a7d578a29 tabs: Fix tabs metric.
bf32270b0 Clean up telemetry/timeline/trace_data*
caee0de0e [TBMv2] Add 'tasks' legacy_unit_info
385d3dd26 Add comments to document that variance/stddev use Bessel's correction
c36ea2490 Remove deprecated trace event name for BlinkSchedulerAsyncSlice.
8fbdf9f10 Fix blink-gc metrics
0043a4a25 [Tracing UI] Fix raster task view
0111e13d0 [Tracing UI] Don't show "View in Picture Debugger" when not available
393b0fd9c [Tracing UI] Use ResizeObserver for picture chart resizing
0ceb1ecce [Tracing UI] Fix drag_handle when there are size constraints
0e9738361 [Tracing] Remove unused resize_sensor.html and css-element-queries
1986f5a95 Change "minutes" legacy unit to timeInMsAutoFormat.
f1e34db09 Fix frame viewer ui issues
8dd790d49 Allow passing timestamps to Timing.mark.
49edbd3a2 Measure TBM2 runMetrics.
9aa395965 [trace-viewer] Update according to chrome side Frame Viewer trace format
5b4ef865e Roll eslint to 4.0.
183509ba6 Shorten normalizedPercentage units to a single decimal digit.
669dcb14a [trace-viewer] Fix "no tree" error in layer quad stack view
a6ccbcf12 [trace-viewer] Fix exception when rasterizing zero-sized picture
bc894f56c [vr] Add post_submit_draw_on_gpu metric
a07c6a933 Histogram - Cache info types
3d2afdf63 [CSMMerging] Refactor breakdown_tree_helper with findEmptyRangesBetweenRanges
075ca3000 Support fuchsia project revisions in tooltips.
b6e96e16f Introduce sequence_manager tracing category.
2048c078d [Dashboard] Fix stdio URLs for histogram uploads
504f52bbe Fix symbolization of heap profiles when so is mapped from apk
86d3e79d8 [Tracing] Android native library resident size statistics.
846cec5c4 HtmlEscape histogram json when rendering results.html
055c03395 Use top-level scheduler tasks in EQT metric.
59aabe8d4 Generate brighter color palette using SinebowColorGenerator.
222de1a4f [TBMv2] Add HAD_FAILURES diagnostic and use it to avoid merging
a22719853 [vr] Increase VR Response UE to 1s
516a69786 HistogramSet - Merge by test paths in add_reserved_diagnostics.
1bae362e7 HistogramSet - Merge by only a subset of tags.
8ddb78550 [Dashboard] Merge histograms by tags in add_reserved_diagnostics
10232e9c9 Histogram - Missing json import in bin/add_reserved_diagnostics
71fd6b902 Revert "Do not add zero values to breakdown diagnostics"
7c43e1be4 Do not add zero values to breakdown diagnostics
1c69b2f42 androidStartupMetric: add first_contentful_paint_time
b5e636db8 HistogramSet - Only merge histograms with stories.
0ad14f9aa [TBMv2] Test add_reserved_diagnostics
72c05e713 Fix frame grouping for [Web]FrameScheduler since it was renamed.
6b8f67cc1 VR: Remove UpdateTexturesAndSizes metric
5d35a2c71 Get navigation info without using FrameLoader snapshots
8b153cc7d Revert "HistogramSet - Only merge histograms with stories."
bffbf166f HistogramSet - Only merge histograms with stories.
85462f12f [TBMv2] Add GenericSet.GetOnlyElement in Python
fe230b673 [Build] Fix tracing Python tests
7e3fcbe35 Reland "More precise self time calculation"
1dfd083b6 Fix sorting results.html
718296623 First set of thread times metrics in TBMv2
17fc85e9c Replace --enable-heap-profiling with --memlog equivalent.
f9a1ae1b3 Fix visual rect support for cc::DisplayItemList
580f1ca28 [Dashboard] Add IS_SUMMARY diagnostic
b3d01529c [fuchsia] Add thread state to fuchsia importer.
1446cf3fe [vr] Fix draw metrics in frameCycleDurationMetric
f73167a68 Ignore i-frame navigationStart events when computing LoadExpectation
4d4ed66a7 Revert "More precise self time calculation"
548a5bc4b androidStartupMetric: add request_start_time
df668c312 Disable failing Snap-It test
27e44012b [Tracing] Fix merge_traces.py script
a0688890f [results.html] Show All by default.
900b947e1 [vr] Add metrics for render step to frameCycleDurationMetric
2210f05b4 More precise self time calculation
875a659c7 Remove the 'deleted' suffix on path name
5c590a9bf Tracing: CPU time computation clean up
b18c0b5df [vr] Fix submit_frame of frameCycleDuration metric
1c706a34f [TBMv2] Allow nulls to be stably stringified
0ac0bb6d1 [TBMv2] Factor mergeHistograms out in merge_histograms_cmdline

Test: ./systrace.py -b 65536 gfx sync sched -t 5
Merged-In: I92804a98fc4fc438f1c5cac79e869a37bebd9416
(cherry picked from commit 367b65a310a72ba207744182de04a89d520b31ee)

Change-Id: I803021272634cf4109b96fdd7f2f5ec93bf5ba51

1 files changed, 248 insertions, 10 deletions

diff --git a/catapult/devil/devil/android/device_utils.py b/catapult/devil/devil/android/device_utils.py
index 5a3db413..518e4393 100644
--- a/catapult/devil/devil/android/device_utils.py
+++ b/catapult/devil/devil/android/device_utils.py
@@ -59,6 +59,29 @@ _DEFAULT_RETRIES = 3
 # the timeout_retry decorators.
 DEFAULT = object()
 
+# A sentinel object to require that calls to RunShellCommand force running the
+# command with su even if the device has been rooted. To use, pass into the
+# as_root param.
+_FORCE_SU = object()
+
+_RECURSIVE_DIRECTORY_LIST_SCRIPT = """
+  function list_subdirs() {
+    for f in "$1"/* ;
+    do
+      if [ -d "$f" ] ;
+      then
+        if [ "$f" == "." ] || [ "$f" == ".." ] ;
+        then
+          continue ;
+        fi ;
+        echo "$f" ;
+        list_subdirs "$f" ;
+      fi ;
+    done ;
+  } ;
+  list_subdirs %s
+"""
+
 _RESTART_ADBD_SCRIPT = """
   trap '' HUP
   trap '' TERM
@@ -88,6 +111,7 @@ _PERMISSIONS_BLACKLIST_RE = re.compile('|'.join(fnmatch.translate(p) for p in [
     'android.permission.DISABLE_KEYGUARD',
     'android.permission.DOWNLOAD_WITHOUT_NOTIFICATION',
     'android.permission.EXPAND_STATUS_BAR',
+    'android.permission.FOREGROUND_SERVICE',
     'android.permission.GET_PACKAGE_SIZE',
     'android.permission.INSTALL_SHORTCUT',
     'android.permission.INJECT_EVENTS',
@@ -649,6 +673,22 @@ class DeviceUtils(object):
         'Version name for %s not found on dumpsys output' % package, str(self))
 
   @decorators.WithTimeoutAndRetriesFromInstance()
+  def GetPackageArchitecture(self, package, timeout=None, retries=None):
+    """Get the architecture of a package installed on the device.
+
+    Args:
+      package: Name of the package.
+
+    Returns:
+      A string with the architecture, or None if the package is missing.
+    """
+    lines = self._GetDumpsysOutput(['package', package], 'primaryCpuAbi')
+    if lines:
+      _, _, package_arch = lines[-1].partition('=')
+      return package_arch.strip()
+    return None
+
+  @decorators.WithTimeoutAndRetriesFromInstance()
   def GetApplicationDataDirectory(self, package, timeout=None, retries=None):
     """Get the data directory on the device for the given package.
 
@@ -670,6 +710,31 @@ class DeviceUtils(object):
     raise device_errors.CommandFailedError(
         'Could not find data directory for %s', package)
 
+  @decorators.WithTimeoutAndRetriesFromInstance()
+  def GetSecurityContextForPackage(self, package, encrypted=False, timeout=None,
+      retries=None):
+    """Gets the SELinux security context for the given package.
+
+    Args:
+      package: Name of the package.
+      encrypted: Whether to check in the encrypted data directory
+          (/data/user_de/0/) or the unencrypted data directory (/data/data/).
+
+    Returns:
+      The package's security context as a string, or None if not found.
+    """
+    directory = '/data/user_de/0/' if encrypted else '/data/data/'
+    for line in self.RunShellCommand(['ls', '-Z', directory],
+                                     as_root=True, check_return=True):
+      split_line = line.split()
+      # ls -Z output differs between Android versions, but the package is
+      # always last and the context always starts with "u:object"
+      if split_line[-1] == package:
+        for column in split_line:
+          if column.startswith('u:object'):
+            return column
+    return None
+
   def TakeBugReport(self, path, timeout=60*5, retries=None):
     """Takes a bug report and dumps it to the specified path.
 
@@ -1064,7 +1129,7 @@ class DeviceUtils(object):
     if run_as:
       cmd = 'run-as %s sh -c %s' % (cmd_helper.SingleQuote(run_as),
                                     cmd_helper.SingleQuote(cmd))
-    if as_root and self.NeedsSU():
+    if (as_root is _FORCE_SU) or (as_root and self.NeedsSU()):
       # "su -c sh -c" allows using shell features in |cmd|
       cmd = self._Su('sh -c %s' % cmd_helper.SingleQuote(cmd))
 
@@ -1202,6 +1267,33 @@ class DeviceUtils(object):
         raise device_errors.CommandFailedError(line, str(self))
 
   @decorators.WithTimeoutAndRetriesFromInstance()
+  def StartService(self, intent_obj, user_id=None, timeout=None, retries=None):
+    """Start a service on the device.
+
+    Args:
+      intent_obj: An Intent object to send describing the service to start.
+      user_id: A specific user to start the service as, defaults to current.
+      timeout: Timeout in seconds.
+      retries: Number of retries
+
+    Raises:
+      CommandFailedError if the service could not be started.
+      CommandTimeoutError on timeout.
+      DeviceUnreachableError on missing device.
+    """
+    # For whatever reason, startservice was changed to start-service on O and
+    # above.
+    cmd = ['am', 'startservice']
+    if self.build_version_sdk >= version_codes.OREO:
+      cmd[1] = 'start-service'
+    if user_id:
+      cmd.extend(['--user', str(user_id)])
+    cmd.extend(intent_obj.am_args)
+    for line in self.RunShellCommand(cmd, check_return=True):
+      if line.startswith('Error:'):
+        raise device_errors.CommandFailedError(line, str(self))
+
+  @decorators.WithTimeoutAndRetriesFromInstance()
   def StartInstrumentation(self, component, finish=True, raw=False,
                            extras=None, timeout=None, retries=None):
     if extras is None:
@@ -1383,7 +1475,7 @@ class DeviceUtils(object):
           missing_dirs.add(posixpath.dirname(d))
 
     if delete_device_stale and all_stale_files:
-      self.RunShellCommand(['rm', '-f'] + all_stale_files, check_return=True)
+      self.RemovePath(all_stale_files, force=True, recursive=True)
 
     if all_changed_files:
       if missing_dirs:
@@ -1483,15 +1575,66 @@ class DeviceUtils(object):
         else:
           to_push.append((host_abs_path, device_abs_path))
       to_delete = device_checksums.keys()
+    # We can't rely solely on the checksum approach since it does not catch
+    # stale directories, which can result in empty directories that cause issues
+    # during copying in efficient_android_directory_copy.sh. So, find any stale
+    # directories here so they can be removed in addition to stale files.
+    if track_stale:
+      to_delete.extend(self._GetStaleDirectories(host_path, device_path))
 
     def cache_commit_func():
-      new_sums = {posixpath.join(device_path, path[len(host_path) + 1:]): val
-                  for path, val in host_checksums.iteritems()}
+      # When host_path is a not a directory, the path.join() call below would
+      # have an '' as the second argument, causing an unwanted / to be appended.
+      if os.path.isfile(host_path):
+        assert len(host_checksums) == 1
+        new_sums = {device_path: host_checksums[host_path]}
+      else:
+        new_sums = {posixpath.join(device_path, path[len(host_path) + 1:]): val
+                    for path, val in host_checksums.iteritems()}
       cache_entry = [ignore_other_files, new_sums]
       self._cache['device_path_checksums'][device_path] = cache_entry
 
     return (to_push, up_to_date, to_delete, cache_commit_func)
 
+  def _GetStaleDirectories(self, host_path, device_path):
+    """Gets a list of stale directories on the device.
+
+    Args:
+      host_path: an absolute path of a directory on the host
+      device_path: an absolute path of a directory on the device
+
+    Returns:
+      A list containing absolute paths to directories on the device that are
+      considered stale.
+    """
+    def get_device_dirs(path):
+      directories = set()
+      command = _RECURSIVE_DIRECTORY_LIST_SCRIPT % cmd_helper.SingleQuote(path)
+      # We use shell=True to evaluate the command as a script through the shell,
+      # otherwise RunShellCommand tries to interpret it as the name of a (non
+      # existent) command to run.
+      for line in self.RunShellCommand(
+          command, shell=True, check_return=True):
+        directories.add(posixpath.relpath(posixpath.normpath(line), path))
+      return directories
+
+    def get_host_dirs(path):
+      directories = set()
+      if not os.path.isdir(path):
+        return directories
+      for root, _, _ in os.walk(path):
+        if root != path:
+          # Strip off the top level directory so we can compare the device and
+          # host.
+          directories.add(
+              os.path.relpath(root, path).replace(os.sep, posixpath.sep))
+      return directories
+
+    host_dirs = get_host_dirs(host_path)
+    device_dirs = get_device_dirs(device_path)
+    stale_dirs = device_dirs - host_dirs
+    return [posixpath.join(device_path, d) for d in stale_dirs]
+
   def _ComputeDeviceChecksumsForApks(self, package_name):
     ret = self._cache['package_apk_checksums'].get(package_name)
     if ret is None:
@@ -1608,6 +1751,8 @@ class DeviceUtils(object):
       except zip_utils.ZipFailedError:
         return False
 
+      logger.info('Pushing %d files via .zip of size %d', len(files),
+                  os.path.getsize(zip_path))
       self.NeedsSU()
       with device_temp_file.DeviceTempFile(
           self.adb, suffix='.zip') as device_temp:
@@ -2279,9 +2424,8 @@ class DeviceUtils(object):
     """
     try:
       ps_cmd = 'ps'
-      # ps behavior was changed in Android above N, http://crbug.com/686716
-      if (self.build_version_sdk >= version_codes.NOUGAT_MR1
-          and self.build_id[0] > 'N'):
+      # ps behavior was changed in Android O and above, http://crbug.com/686716
+      if self.build_version_sdk >= version_codes.OREO:
         ps_cmd = 'ps -e'
       if pattern:
         return self._RunPipedShellCommand(
@@ -2331,6 +2475,29 @@ class DeviceUtils(object):
       processes.append(ProcessInfo(**row))
     return processes
 
+  def _GetDumpsysOutput(self, extra_args, pattern=None):
+    """Runs |dumpsys| command on the device and returns its output.
+
+    This private method implements support for filtering the output by a given
+    |pattern|, but does not do any output parsing.
+    """
+    try:
+      cmd = ['dumpsys'] + extra_args
+      if pattern:
+        cmd = ' '.join(cmd_helper.SingleQuote(s) for s in cmd)
+        return self._RunPipedShellCommand(
+            '%s | grep -F %s' % (cmd, cmd_helper.SingleQuote(pattern)))
+      else:
+        cmd = ['dumpsys'] + extra_args
+        return self.RunShellCommand(cmd, check_return=True, large_output=True)
+    except device_errors.AdbShellCommandFailedError as e:
+      if e.status and isinstance(e.status, list) and not e.status[0]:
+        # If dumpsys succeeded but grep failed, there were no lines matching
+        # the given pattern.
+        return []
+      else:
+        raise
+
   # TODO(#4103): Remove after migrating clients to ListProcesses.
   @decorators.WithTimeoutAndRetriesFromInstance()
   def GetPids(self, process_name=None, timeout=None, retries=None):
@@ -2438,6 +2605,30 @@ class DeviceUtils(object):
         check_return=True)
 
   @decorators.WithTimeoutAndRetriesFromInstance()
+  def SetWebViewImplementation(self, package_name, timeout=None, retries=None):
+    """Select the WebView implementation to the specified package.
+
+    Args:
+      package_name: The package name of a WebView implementation. The package
+        must be already installed on the device.
+      timeout: timeout in seconds
+      retries: number of retries
+
+    Raises:
+      CommandFailedError on failure.
+      CommandTimeoutError on timeout.
+      DeviceUnreachableError on missing device.
+    """
+    output = self.RunShellCommand(
+        ['cmd', 'webviewupdate', 'set-webview-implementation', package_name],
+        single_line=True, check_return=True)
+    if output == 'Success':
+      logging.info('WebView provider set to: %s', package_name)
+    else:
+      raise device_errors.CommandFailedError(
+          'Error setting WebView provider: %s' % output, str(self))
+
+  @decorators.WithTimeoutAndRetriesFromInstance()
   def TakeScreenshot(self, host_path=None, timeout=None, retries=None):
     """Takes a screenshot of the device.
 
@@ -2615,7 +2806,7 @@ class DeviceUtils(object):
 
   @classmethod
   def HealthyDevices(cls, blacklist=None, device_arg='default', retry=True,
-                     **kwargs):
+                     abis=None, **kwargs):
     """Returns a list of DeviceUtils instances.
 
     Returns a list of DeviceUtils instances that are attached, not blacklisted,
@@ -2639,6 +2830,8 @@ class DeviceUtils(object):
               blacklisted.
       retry: If true, will attempt to restart adb server and query it again if
           no devices are found.
+      abis: A list of ABIs for which the device needs to support at least one of
+          (optional).
       A device serial, or a list of device serials (optional).
 
     Returns:
@@ -2674,14 +2867,23 @@ class DeviceUtils(object):
         return True
       return False
 
+    def supports_abi(abi, serial):
+      if abis and abi not in abis:
+        logger.warning("Device %s doesn't support required ABIs.", serial)
+        return False
+      return True
+
     def _get_devices():
       if device_arg:
         devices = [cls(x, **kwargs) for x in device_arg if not blacklisted(x)]
       else:
         devices = []
         for adb in adb_wrapper.AdbWrapper.Devices():
-          if not blacklisted(adb.GetDeviceSerial()):
-            devices.append(cls(_CreateAdbWrapper(adb), **kwargs))
+          serial = adb.GetDeviceSerial()
+          if not blacklisted(serial):
+            device = cls(_CreateAdbWrapper(adb), **kwargs)
+            if supports_abi(device.GetABI(), serial):
+              devices.append(device)
 
       if len(devices) == 0 and not allow_no_devices:
         raise device_errors.NoDevicesError()
@@ -2806,3 +3008,39 @@ class DeviceUtils(object):
       return
     self.SendKeyEvent(keyevent.KEYCODE_POWER)
     timeout_retry.WaitFor(screen_test, wait_period=1)
+
+  @decorators.WithTimeoutAndRetriesFromInstance()
+  def ChangeOwner(self, owner_group, paths, timeout=None, retries=None):
+    """Changes file system ownership for permissions.
+
+    Args:
+      owner_group: New owner and group to assign. Note that this should be a
+        string in the form user[.group] where the group is option.
+      paths: Paths to change ownership of.
+
+      Note that the -R recursive option is not supported by all Android
+      versions.
+    """
+    if not paths:
+      return
+    self.RunShellCommand(['chown', owner_group] + paths, check_return=True)
+
+  @decorators.WithTimeoutAndRetriesFromInstance()
+  def ChangeSecurityContext(self, security_context, paths, timeout=None,
+                            retries=None):
+    """Changes the SELinux security context for files.
+
+    Args:
+      security_context: The new security context as a string
+      paths: Paths to change the security context of.
+
+      Note that the -R recursive option is not supported by all Android
+      versions.
+    """
+    if not paths:
+      return
+    command = ['chcon', security_context] + paths
+
+    # Note, need to force su because chcon can fail with permission errors even
+    # if the device is rooted.
+    self.RunShellCommand(command, as_root=_FORCE_SU, check_return=True)