toolchain-utils: provide option to migrate nightly tests to recipe

This patch provides two new options to buildbot_test_toolchain script,
which launches nightly performance tests for toolchain.

Option `--recipe` introduces new location and mechanism to find latest
image from recipe bucket. The recipe builder for each board monitored by
toolchain team builds an image every day and store results to certain gs
locaton. When this option specified, the script no longer launches
tryjob to get a new image but tries to access the lastest existing image
in the last two days from the gs location.

Option `--test` makes it easier to test locally, it blocks all locations
in chrotomation machine and do not send or store unnecessary results
after tests finish.

A small change to lock_machine.py fixes an issue while related to
swarming when testing this change.

BUG=chromium:1071271
TEST=./buildbot_test_toolchains.py --board=lulu
     --remote=chromeos2-row9-rack8-host7.cros
     --chromeos_root=$chromiumos --test=True --recipe=True

Change-Id: I724a31ce174955c1f002bb7e330919f23535e105
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2151717
Reviewed-by: Tiancong Wang <tcwang@google.com>
Tested-by: Zhizhou Yang <zhizhouy@google.com>
Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
Commit-Queue: Zhizhou Yang <zhizhouy@google.com>

1 files changed, 33 insertions, 1 deletions

diff --git a/cros_utils/buildbot_utils.py b/cros_utils/buildbot_utils.py
index 859e0134..e0c54782 100644
--- a/cros_utils/buildbot_utils.py
+++ b/cros_utils/buildbot_utils.py
@@ -243,8 +243,10 @@ def GetLatestImage(chromeos_root, path):
 
   ce = command_executer.GetCommandExecuter()
   command = ('gsutil ls gs://chromeos-image-archive/%s' % path)
-  _, out, _ = ce.ChrootRunCommandWOutput(
+  ret, out, _ = ce.ChrootRunCommandWOutput(
       chromeos_root, command, print_to_console=False)
+  if ret != 0:
+    raise RuntimeError('Failed to list buckets with command: %s.' % command)
   candidates = [l.split('/')[-2] for l in out.split()]
   candidates = [fmt.match(c) for c in candidates]
   candidates = [[int(r) for r in m.group(1, 2, 3, 4)] for m in candidates if m]
@@ -258,3 +260,33 @@ def GetLatestImage(chromeos_root, path):
 
     if DoesImageExist(chromeos_root, build):
       return build
+
+
+def GetLatestRecipeImage(chromeos_root, path):
+  """Get latest nightly test image from recipe bucket.
+
+  Image location example:
+  $ARCHIVE/lulu-llvm-next-nightly/R84-13037.0.0-31011-8883172717979984032
+  """
+
+  fmt = re.compile(r'R([0-9]+)-([0-9]+).([0-9]+).([0-9]+)-([0-9]+)')
+
+  ce = command_executer.GetCommandExecuter()
+  command = ('gsutil ls gs://chromeos-image-archive/%s' % path)
+  ret, out, _ = ce.ChrootRunCommandWOutput(
+      chromeos_root, command, print_to_console=False)
+  if ret != 0:
+    raise RuntimeError('Failed to list buckets with command: %s.' % command)
+  candidates = [l.split('/')[-2] for l in out.split()]
+  candidates = [(fmt.match(c), c) for c in candidates]
+  candidates = [([int(r)
+                  for r in m[0].group(1, 2, 3, 4, 5)], m[1])
+                for m in candidates
+                if m]
+  candidates.sort(key=lambda x: x[0], reverse=True)
+  # Try to get ony last two days of images since nightly tests are run once
+  # another day.
+  for c in candidates[:2]:
+    build = '%s/%s' % (path, c[1])
+    if DoesImageExist(chromeos_root, build):
+      return build