Download official & trybot images; treat as local images for tests.

Currently we cannot get md5sums for official & trybot images, which
causes multiple problems:  Crosperf re-images the DUT every time it
runs a test for an image, even if the image is already on the DUT;
if crosperf is running an experiment with multiple official or trybot
images, it can get confused about which results belong to which image;
caching does not work properly without md5sums.

To fix all of these problems, this CL changes Crosperf to download the
official or trybot images into the /tmp directory in the chroot, and from
there it can treat  them as local images (getting md5sums etc).  In order
to download them, it first has to translate the xbuddy syntax (which can
contain aliases such as 'lumpy/dev-latest') into the actual image name
(e.g. lumpy-release/R36-5727.0.0).  This translation is done by a new
script, translate_xbuddy.py, which must be run from the toolchain-utils
directory inside the chroot.

BUG=356279,356474,356476
TEST=Tested with official and trybot images, using both correct
designations and xbuddy aliases; tested with the image missing from
the chroot and with it already in /tmp; tested with and without caching.
Also tested with the experiment file included in the 3 bugs.

Change-Id: I917b5520604b7d4851db2c8123165e81b866da2b
Reviewed-on: https://chrome-internal-review.googlesource.com/159465
Reviewed-by: Yunlian Jiang <yunlian@google.com>
Commit-Queue: Caroline Tice <cmtice@google.com>
Tested-by: Caroline Tice <cmtice@google.com>

1 files changed, 52 insertions, 0 deletions

diff --git a/crosperf/download_images.py b/crosperf/download_images.py
new file mode 100644
index 00000000..ac8fd3a4
--- /dev/null
+++ b/crosperf/download_images.py
@@ -0,0 +1,52 @@
+#!/usr/bin/python
+
+# Copyright (c) 2014 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import os
+
+from utils import command_executer
+
+class ImageDownloader(object):
+
+  def __init__(self, logger_to_use=None, log_level="verbose"):
+    self._logger = logger_to_use
+    self.log_level = log_level
+    self._ce = command_executer.GetCommandExecuter(self._logger,
+                                                   log_level = self.log_level)
+
+  def Run(self, chromeos_root, xbuddy_label):
+    # Get the translation of the xbuddy_label into the real Google Storage
+    # image name.
+    command = ("cd ~/trunk/src/third_party/toolchain-utils/crosperf; "
+               "python translate_xbuddy.py '%s'" % xbuddy_label)
+    retval, build_id_tuple_str, _ = self._ce.ChrootRunCommand(chromeos_root,
+                                                          command, True)
+    build_id_tuple = eval(build_id_tuple_str)
+    build_id = build_id_tuple[0]
+
+    # Make sure the directory for downloading the image exists.
+    download_path = os.path.join(chromeos_root, "chroot/tmp",
+                                 build_id)
+    image_path = os.path.join(download_path, "chromiumos_test_image.bin")
+    if not os.path.exists(download_path):
+      command = "mkdir -p %s" % download_path
+      self._ce.RunCommand (command)
+
+    # Check to see if the image has already been downloaded.  If not,
+    # download the image.
+    retval = 0
+    if not os.path.exists(image_path):
+      command = ("gsutil cp gs://chromeos-image-archive/%s"
+                 "/chromiumos_test_image.tar.xz /tmp/%s" % (build_id,
+                                                            build_id))
+
+      retval = self._ce.ChrootRunCommand(chromeos_root, command)
+
+      # Uncompress and untar the downloaded image.
+      command = ("cd /tmp/%s ;unxz chromiumos_test_image.tar.xz; "
+                 "tar -xvf chromiumos_test_image.tar" % build_id)
+      retval = self._ce.ChrootRunCommand(chromeos_root, command)
+
+    return retval, image_path