diff options
| author | cmtice <cmtice@google.com> | 2014-07-14 15:29:02 -0700 |
|---|---|---|
| committer | chrome-internal-fetch <chrome-internal-fetch@google.com> | 2014-07-16 07:08:04 +0000 |
| commit | 74b3d3234dc857ebee3af81faa3f8f420ac6ecce (patch) | |
| tree | 13b70df9f058bfaf0f006e76b67aa2e3b1b90e45 /crosperf | |
| parent | c490e073c811f60d6fdfec5c193e7f042302b5e8 (diff) | |
| download | toolchain-utils-74b3d3234dc857ebee3af81faa3f8f420ac6ecce.tar.gz | |
Fix issue where Telemetry tests appear to pass, but really failed
Occasionally Telemetry tests will return a "pass" even though the test
did not run (e.g. if the test was disabled). Crosperf was looking for
results for every passing test and dying when it failed to find a
result for these false passes. This CL looks for the false pass and
changes it fail, as well as writing out a warning message.
A bug was filed for this in Buganizer:
https://b.corp.google.com/issue?id=16190266, but since this was not
filed as a Chrome/ChromeOS bug, I will leave the bug mark below as
None (to avoid confusing the bug droid).
BUG=None
TEST=Tested this against a Telemetry run with both false and real
passes, and it worked properly.
Change-Id: Ie010c0cd5b0775d0e20cfd6967975eb4e768c55c
Reviewed-on: https://chrome-internal-review.googlesource.com/169115
Reviewed-by: Yunlian Jiang <yunlian@google.com>
Commit-Queue: Caroline Tice <cmtice@google.com>
Tested-by: Caroline Tice <cmtice@google.com>
Diffstat (limited to 'crosperf')
| -rw-r--r-- | crosperf/results_report.py | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/crosperf/results_report.py b/crosperf/results_report.py index 2f352fe8..e74e3b4b 100644 --- a/crosperf/results_report.py +++ b/crosperf/results_report.py @@ -97,11 +97,26 @@ class ResultsReport(object): cell.header = True return [[cell]] + def _FixFalsePositiveTests(self, result, table_type): + # Occasionally Telemetry tests will not fail but they will not return a + # result, either. Look for those cases, and force them to be a fail. + # (This can happen if, for example, the test has been disabled.) + for k in result: + for run in result[k]: + run_dict = run[0] + if len(run_dict) != 1 or run_dict['retval'] != 0: + continue + run_dict['retval'] = 1 + if table_type == 'summary': + print ("WARNING: Test '%s' appears to have succeeded but returned" + " no results." % k) + def _GetTables(self, labels, benchmark_runs, columns, table_type): tables = [] ro = ResultOrganizer(benchmark_runs, labels, self.benchmarks) result = ro.result label_name = ro.labels + self._FixFalsePositiveTests(result, table_type) for item in result: runs = result[item] for benchmark in self.benchmarks: |