crosperf: Make results_report more general.

The goal of this patch is to allow us to use results_report to generate
non-ChromeOS-specific results reports. It's meant to be a nop for
regular crosperf runs (except that, in HTML report generation,
whitespace in the HTML itself will look a bit different.)

Moreover, results_report used to shuffle Experiments around, each of
which, unsurprisingly, contained *tons* of data about an experiment.
So, part of this patch was reducing results_report's reliance on
Experiments, in favor of a new type, BenchmarkResult.

Some parts of results_report still rely on Experiments, but only to
provide *extra* data. The minimum amount of data needed to make a
results report is contained in a BenchmarkResult, and there's a
convenient API we can use to make a BenchmarkResult out of an
Experiment.

This patch also does a massive refactor of results_report, because lots
of the code was mildly more icky than it is with this patch applied.
The refactor-for-prettiness and refactor-for-BenchmarkResults kind of
go hand-in-hand, so it's really hard to split them out. The only part
that's not so difficult to split out is the refactor to
results_report_templates, but the value of splitting that out is
questionable (to me).

Speaking of which, all HTML magicks that were in results_report is now
in results_report_templates, and now use *actual* Templates. It -- and
HTMLReportGenerator -- are hopefully more readable as a result.

Next, this makes JSONRsultsReport's GetReport() return a string
containing JSON data, rather than calling a callback to write
that string to a file.

Finally, this includes a change to perf_table. Namely, its removal. It
was otherwise unused, and I couldn't get it to even work, so I cleaned
it up, made a perf report parser (because I couldn't get the cros_utils
one to work, either...), and made sure it functions. If we're able to
produce perf reports that cros_utils can parse, I'm happy to back my
new parser out; I was primarily using it so I could be sure I didn't
break HTML report generation.

BUG=chromium:641098
TEST=./run_tests.sh passes

Change-Id: I437de9eb39e00c9dd5c223ecd27feaaab544a6fd
Reviewed-on: https://chrome-internal-review.googlesource.com/282217
Commit-Ready: George Burgess <gbiv@google.com>
Tested-by: George Burgess <gbiv@google.com>
Reviewed-by: Caroline Tice <cmtice@google.com>

1 files changed, 27 insertions, 7 deletions

diff --git a/crosperf/experiment_runner.py b/crosperf/experiment_runner.py
index 542a0a60..b30c8bd5 100644
--- a/crosperf/experiment_runner.py
+++ b/crosperf/experiment_runner.py
@@ -1,5 +1,9 @@
-# Copyright 2011-2015 Google Inc. All Rights Reserved.
+# Copyright (c) 2011 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
 """The experiment runner module."""
+from __future__ import print_function
+
 import getpass
 import os
 import shutil
@@ -22,6 +26,19 @@ from results_report import TextResultsReport
 from results_report import JSONResultsReport
 from schedv2 import Schedv2
 
+def _WriteJSONReportToFile(experiment, results_dir, json_report):
+  """Writes a JSON report to a file in results_dir."""
+  has_llvm = any('llvm' in l.compiler for l in experiment.labels)
+  compiler_string = 'llvm' if has_llvm else 'gcc'
+  board = experiment.labels[0].board
+  filename = 'report_%s_%s_%s.%s.json' % (
+      board, json_report.date, json_report.time.replace(':', '.'),
+      compiler_string)
+  fullname = os.path.join(results_dir, filename)
+  report_text = json_report.GetReport()
+  with open(fullname, 'w') as out_file:
+    out_file.write(report_text)
+
 
 class ExperimentRunner(object):
   """ExperimentRunner Class."""
@@ -193,7 +210,7 @@ class ExperimentRunner(object):
         self._UnlockAllMachines(experiment)
 
   def _PrintTable(self, experiment):
-    self.l.LogOutput(TextResultsReport(experiment).GetReport())
+    self.l.LogOutput(TextResultsReport.FromExperiment(experiment).GetReport())
 
   def _Email(self, experiment):
     # Only email by default if a new run was completed.
@@ -211,11 +228,11 @@ class ExperimentRunner(object):
       label_names.append(label.name)
     subject = '%s: %s' % (experiment.name, ' vs. '.join(label_names))
 
-    text_report = TextResultsReport(experiment, True).GetReport()
+    text_report = TextResultsReport.FromExperiment(experiment, True).GetReport()
     text_report += ('\nResults are stored in %s.\n' %
                     experiment.results_directory)
     text_report = "<pre style='font-size: 13px'>%s</pre>" % text_report
-    html_report = HTMLResultsReport(experiment).GetReport()
+    html_report = HTMLResultsReport.FromExperiment(experiment).GetReport()
     attachment = EmailSender.Attachment('report.html', html_report)
     email_to = experiment.email_to or []
     email_to.append(getpass.getuser())
@@ -237,14 +254,17 @@ class ExperimentRunner(object):
 
     self.l.LogOutput('Storing results report in %s.' % results_directory)
     results_table_path = os.path.join(results_directory, 'results.html')
-    report = HTMLResultsReport(experiment).GetReport()
+    report = HTMLResultsReport.FromExperiment(experiment).GetReport()
     if self.json_report:
-      JSONResultsReport(experiment).GetReport(results_directory)
+      json_report = JSONResultsReport.FromExperiment(experiment,
+                                                     json_args={'indent': 2})
+      _WriteJSONReportToFile(experiment, results_directory, json_report)
+
     FileUtils().WriteFile(results_table_path, report)
 
     self.l.LogOutput('Storing email message body in %s.' % results_directory)
     msg_file_path = os.path.join(results_directory, 'msg_body.html')
-    text_report = TextResultsReport(experiment, True).GetReport()
+    text_report = TextResultsReport.FromExperiment(experiment, True).GetReport()
     text_report += ('\nResults are stored in %s.\n' %
                     experiment.results_directory)
     msg_body = "<pre style='font-size: 13px'>%s</pre>" % text_report