cidb: use the cidb on our builders

This CL enables use of the cidb on builders run with --buildbot. In
particular:
 - The cidb connection factory uses a cached cidb connection to avoid
   creating duplicate connections per cbuildbot execution.
 - A new stage, BuildStart is added, which is the first stage to run and
   which inserts a build row into the database if appropriate.
 - ReportBuildStart is renamed to BuildReexecutionFinished for clarity.
 - UploadMetadata is extended to also update the build row in the
   database.
 - Calls to RecordCLAction are accompanied by a CL action database
   insert.
 - ReportStage calls FinishBuild and writes the final build status to
   the database.

BUG=chromium:387755
TEST=cidb_integration_test;
remote trybots with --buildbot --debug, verified by inspecting database
that expected data is written there.

Change-Id: Ia2d9d355a744cafda0ade851be141519c56093d8
Reviewed-on: https://chromium-review.googlesource.com/211648
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>

1 files changed, 13 insertions, 4 deletions

diff --git a/cbuildbot/metadata_lib.py b/cbuildbot/metadata_lib.py
index ef07da58f..72319a274 100644
--- a/cbuildbot/metadata_lib.py
+++ b/cbuildbot/metadata_lib.py
@@ -234,6 +234,19 @@ class CBuildbotMetadata(object):
     temp['board-metadata'] = per_board_dict
     return temp
 
+  # TODO(akeshet): crbug.com/406522 special case cl_actions and board-metadata
+  # so that GetValue can work with them as well.
+  def GetValue(self, key):
+    """Get an item from the metadata dictionary.
+
+    This method is in most cases an inexpensive equivalent to:
+    GetDict()[key]
+
+    However, it cannot be used for items like 'cl_actions' or 'board-metadata'
+    which are not stored directly in the metadata dictionary.
+    """
+    return self._metadata_dict[key]
+
   def GetJSON(self):
     """Return a JSON string representation of metadata."""
     return json.dumps(self.GetDict())
@@ -463,10 +476,6 @@ class BuildData(object):
       if bd.build_number is None:
         cros_build_lib.Warning('Metadata at %s was missing build number.',
                                url)
-        # metadata.json can be missing a build number if the build died before
-        # ReportBuildStartStage. See crbug.com/369748. As a workaround so that
-        # gather_builder_stats can still record these builds in the spreadsheet
-        # try to infer the build number from the file's url.
         m = re.match(r'.*-b([0-9]*)/.*', url)
         if m:
           inferred_number = int(m.groups()[0])