crosperf: set recommended iterations for benchmarks

This CL associates estimated standard deviations to each benchmark,
according to experiments. The recommended iterations can be specified by
setting iterations = 0 in the experiment files. Setting it to numbers
greater than 0 will override the default iterations and behaves exactly
the same as before.

With this change, benchmarks in all_toolchain_perf get no more than 2%
margin of error within 90% of time. See crbug.com/673558 for how the
standard deviations are estimated.

BUG=chromium:673558
TEST=all_toolchain_perf + page_cycler_v2.typical_25 finishes in 3.5
     hours for an image on chell.

Change-Id: Ie2ed07878c1237ad31a8568ae3fd3fb96cd11f3b
Reviewed-on: https://chromium-review.googlesource.com/424915
Commit-Ready: Ting-Yuan Huang <laszio@chromium.org>
Tested-by: Ting-Yuan Huang <laszio@chromium.org>
Reviewed-by: Caroline Tice <cmtice@chromium.org>

1 files changed, 2 insertions, 2 deletions

diff --git a/buildbot_test_toolchains.py b/buildbot_test_toolchains.py
index 5bef28a4..028d7f1a 100755
--- a/buildbot_test_toolchains.py
+++ b/buildbot_test_toolchains.py
@@ -175,12 +175,12 @@ class ToolchainComparator(object):
     experiment_tests = """
     benchmark: all_toolchain_perf {
       suite: telemetry_Crosperf
-      iterations: 3
+      iterations: 0
     }
 
     benchmark: page_cycler_v2.typical_25 {
       suite: telemetry_Crosperf
-      iterations: 2
+      iterations: 0
       run_local: False
       retries: 0
     }