aboutsummaryrefslogtreecommitdiff
path: root/crosperf/experiment_factory.py
AgeCommit message (Collapse)Author
2020-02-07crosperf: introduce new test_that and skylab run for suite_runnerZhizhou Yang
With client tests wrapped up by crosperf_Wrapper server tests, we need a categorizing for all tests lunched by suite runner. BUG=chromium:984790 TEST=Passed unittest and simple experiment. Change-Id: Ib7dc848cf5c18c3d79db059163ed1217fd7d318f Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1977040 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-12-04crosperf: move enable_aslr and turbostat option into dut_configZhizhou Yang
This patch moves enable_aslr and turbostat into dut_config dictionary, since they are all device setup steps, and it make code cleaner and easier to migrate to telemetry_Crosperf. TEST=Passed unittests. BUG=chromium:1020655 Change-Id: I29a649e76591e206efc19bb2b8a8df9aead16575 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1947903 Reviewed-by: Denis Nikitin <denik@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-10-27crosperf: Add option to not download debug symbolsZhizhou Yang
This patch introduces a new option --download_debug to crosperf. We will only download debug symbols when this option is set to True. This will save space when perf test users doesn't need it. TEST=passed unittests. BUG=chromium:1010294 Change-Id: Ic80958377f1c25cb04e3b7b56b40fafcada6d2d8 Reviewed-on: https://chromium-review.googlesource.com/1865958 Tested-by: Zhizhou Yang <zhizhouy@google.com> Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org>
2019-10-15toolchain-utils: change naming related to AFE lockZhizhou Yang
After introducing the new locking mechanism, the behavior of AFELockManager totally changed. This file changed the naming of all related code in toochain-utils. This patch also changed the behavior of locks_dir option, deprecated the use_file_lock option explicitly. TEST=Tested with different DUT types. BUG=chromium:1006434 Change-Id: Ib15efce54ec4d4c5c2a18fecca3f350248462035 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1863530 Reviewed-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-10-09crosperf: Change video crosbolt benchmark to Tast versionKeiichi Watanabe
Since video autotests are already ported to Tast and stopped running, replace crosbolt benchmark with Tast version. BUG=chromium:953553 TEST=None Change-Id: I09a44027ab2dc338679d90f4aa8ad362f160852e Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1801410 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2019-09-24crosperf: Add top_interval argument to experimentDenis Nikitin
Added the following optional argument in the global settings: "top_interval" - Run top command in the background of a benchmark with interval of sampling specified in seconds (float type). With zero value don't run top (default value). Top output is stored in <resultsdir>/top.log. BUG=chromium:966514 TEST=unittests and HW tests on eve, kevin64, scarlet passed. Change-Id: I87b187c85912140fc0c16e224580b60061c8bd94 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1817121 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-16crosperf: Setup CPU frequencyDenis Nikitin
Added "cpu_freq_pct" in global settings specifying percent of the max frequency which will be set on DUT during benchmark run. The actual value will be chosen from available frequencies less than or equal to the calculated number. Feature requires scaling_available_frequencies. On the platforms not supporting scaling_available_frequencies "cpu_freq_pct" will be ignored. BUG=chromium:966514 TEST=Unittest and local HW tests on veyron, scarlet and kevin64 passed. Change-Id: I175e2836ecf02fcf6b93d196926c7801d7f016c0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1803780 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-09-10crosperf: Add dut config arguments to experimentDenis Nikitin
In global settings added optional arguments: "cooldown_time" - wait time prior running a benchmark (default: 0), "cooldown_temp" - temperature threshold for waiting (default: 40), "governor" - CPU governor (default: performance), "cpu_usage" - Restrict CPU usage to specific configurations (default: all). "turbostat" argument is moved from benchmark to global settings. Current CL does not apply configurations. Instead it just propagates parameters to SuiteRunner class through "dut_config" dictionary. BUG=chromium:966514 TEST=Unitest and local HW tests passed. Change-Id: I1b2a65883e5176fdde49c9858ebe62a097df89cb Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1778515 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-26crosperf: Add jetstream2 to telemetry perf tests.Tiancong Wang
As jetstream2 is added to telemetry benchmarks, we want to add it to the list of telemetry toolchain perf tests, and add supports in crosperf. Also fixed some formatting problems of the file. BUG=None TEST=jetstream2 run locally, will be tested more on nightly tests. Change-Id: I89f748ca8d4a194406688733e91cca47f8a3f5fe Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1769482 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-07toolchain-utils: Fix argument-passing error in experiment_factory.pyCaroline Tice
Fix all calls to Benchmark inside experiment_factory.py to pass all args, to prevent an arg with a default value accidentally being passed a value for a different arg. BUG=chromium:991396 TEST=Tested locally with Crosperf and it worked properly. Change-Id: Ia8096e2989e179723e122090ea40a11747070b22 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1740646 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Tested-by: Caroline Tice <cmtice@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
2019-08-02crosperf: Add "intel_pstate" and "turbostat" argsDenis Nikitin
In global setting added "intel_pstate" option. Type: string. Values: active, passive, no_hwp, ''(equivalent to active). Default ''. In benchmark setting added "turbostat" option. Type: boolean. Default True. BUG=chromium:966514 TEST=tested on samus, eve, rammus Change-Id: I1533c6d6586aab5acf3665bce68920a0ab3092bc Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1726635 Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: Sean Abraham <seanabraham@chromium.org> Reviewed-by: Sean Abraham <seanabraham@chromium.org> Tested-by: Sean Abraham <seanabraham@chromium.org>
2019-08-01crosperf: check skylab tool exists or not for skylab modeZhizhou Yang
This patch checks whether skylab tool is installed when skylab mode is specified. After installing it, it requires user to login the tool manually for the first time since there is no way to do it automatically. BUG=chromium:984790 TEST=Tested locally and passed unittest. Change-Id: Iff03ed0552127837264d8437e2b73346f1abdbd4 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1727264 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-07-25Toolchain-utils: update crosbolt listJian Cai
Move loading.desktop and rendering.desktop from crosbolt_perf_tests to telemetry_crosbolt_perf_tests. Otherwise test_that does not pass correct suite type. BUG=chromium:970061 TEST=local tests Change-Id: I485335fd285d0e132e0d7a2fe309a6303931de56 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1717127 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Jian Cai <jiancai@google.com>
2019-07-23toolchain-utils: update crosbolt benchmarksJian Cai
Current crosbolt benchmarks we run are out-of-date. Update the list to keep up with go/crosbolt. BUG=chromium:970061 TEST=localtests Change-Id: Ib572a68257a23cc32d7211bfbfbf4cebfd3f89c0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1700150 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Jian Cai <jiancai@google.com>
2019-07-22crosperf: introduce skylab field and pass to experiment and labelZhizhou Yang
This is the initial patch to migrate crosperf from autotest to skylab for lab runs. This patch introduces skylab field in experiment, parsed it in settings_factory and restructured label class and experiment class so that further steps can use skylab field. TEST=Passed all unit tests BUG=chromium:984790 Change-Id: I1104fea739977c522938790012f4f3f522f9d65f Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1705234 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-04-18crosperf: generate perf report with correct debug filesZhizhou Yang
This patch fixes the issue in chromium:946588. This patch makes perf report no longer use hard code debug directories. There are several different situations: 1) When running tests on a downloaded image, it will download debug.tgz from gs, extract it to debug_files in /tmp. Options --symfs and --vmlinux will depend on this directory, and throw a warning to user that --kallsyms cannot be applied. 2) If running with downloaded image and debug.tgz could not work, then we will try to use local build, but give user a warning that it may not match real symbols well. 3) When running tests with local build, try to find debug info from /build/$board directory. Thus, this patch added a new field in label, called 'debug_path', if this is manually set in experiment file, then crosperf will directly use the location. Downloading of debug.tgz will only happen when perf_args is set in global settings. TEST=Passed all unit tests, tested with eve and sand. BUG=chromium:946588 Change-Id: I7f35d1216d912c8526d5501748f951face1273aa Reviewed-on: https://chromium-review.googlesource.com/1561780 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-03-15crosperf: Feature to ignore min and max value in resultsZhizhou Yang
This patch provides an option for user to ignore min and max value in the results when generating report. User can use this feature by specifying `ignore_min_max: True` (which by default is False) in experiment file. When values count in a single test is smaller than 3, we automatically ignore this option for it, and print out a warning. BUG=chromium:938758 TEST=Tested with examples in general/cwp mode report generation. Passed all unit tests. Change-Id: I36a4c4d99836c201cdd2f2f9f2a4b1a4ffdaa47d Reviewed-on: https://chromium-review.googlesource.com/1521054 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2019-02-12crosperf: Disable ASLR in crosperf by defaultTiancong Wang
Disabling Address Space Layout Randomization (ASLR) can reduce much variance when measuring performance of benchmark from run to run. This CL will disable ASLR in crosperf by default and add a flag '--enable_aslr' for users to enable it for other testing purpose. BUG=chromium:930332 TEST=Tested locally for several boards and several benchmarks Change-Id: Icfbe477a99ef9a68b503e4979cf2ff47d07a128f Reviewed-on: https://chromium-review.googlesource.com/1398522 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Tiancong Wang <tcwang@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-02-08crosperf: in cwp mode, first check whether iterations are the sameZhizhou Yang
In cwp mode, we ask user to set same iterations for different benchmark runs, but if user did not do so, he will spend too long running all benchmarks and then get an error message in the end. This patch checks whether iterations in experiment file are the same at the time we construct benchmark. Also the patch fixed a small test case mismatch in weight check so that all unit tests can pass. Fixed format issues from pylint. BUG=chromium:929390 TEST=All unit tests passed. Change-Id: I0eae51205e8c0fa80131c5b79c4773e88c5b0915 Reviewed-on: https://chromium-review.googlesource.com/1459854 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2019-01-24crosperf: allow weight in cwp approximation larger than 1Zhizhou Yang
This patch changed the check on the weight in cwp approximation mode. The weight should be a float no less than 0. BUG=chromium:924593 TEST=Tested with a simple experiment on eve. Change-Id: Ia5ef804b5f11e36f6b5066fba6f40855bd716d2f Reviewed-on: https://chromium-review.googlesource.com/1431272 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2018-12-06crosperf: make cwp_dso only work for run_local=FalseZhizhou Yang
We do not want to run cwp_dso experiment with run_local to be True. Note that run_local is set to True by default. BUG=chromium:902785 TEST=Passed all unit tests Change-Id: I11688f34b3c96ac91d3a75ec645bdb5430bef69e Reviewed-on: https://chromium-review.googlesource.com/1363472 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-11-16crosperf: Passing dso and weight to result_cache.pyZhizhou Yang
To generate and collect dso sample count, this patch passes dso and weight into result_cache.py, by setting them as members of Benchmark object. Meantime, fixed all related unit tests. BUG=chromium:902785 TEST=Passed all unit tests and sample tests locally. Change-Id: I483a00d199d48fd5e77d882d7ac81bf93b6940df Reviewed-on: https://chromium-review.googlesource.com/1338344 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-11-15crosperf: do not allow more than one story tags in test_argsZhizhou Yang
This patch added a check at stage of parsing benchmark, to ensure that story tag only appear once in test_args. This is because more than one story tags will cause an implicit failure when running benchmark on DUT and crosperf cannot catch it. BUG=chromium:902466 TEST=Passed unit tests and a simple test Change-Id: Ic8e4aa19928911fb4ed03529ab89471856cec05c Reviewed-on: https://chromium-review.googlesource.com/1330010 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-11-13crosperf: introduce cwp_dso and weight for CWP approximationZhizhou Yang
This patch does initial work to support CWP approximation in crosperf. It introduceds a global option cwp_dso, which is used to specify the type of DSO to use for approximation; an option weight in benchmark setting, which decides the weight of each benchmark in test. This patch also contains some value checks for these two options. BUG=chromium:902785 TEST=Passed all unit tests and sample tests locally. Change-Id: I565c4baf0630ce6c1b62ad0398d05a5f336aac6e Reviewed-on: https://chromium-review.googlesource.com/1323473 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-11-09Crosperf: Treat benchmarks with story- test_args as different benchmarksZhizhou Yang
The bug refers to https://crbug.com/902466. This patch modified the behavior of crosperf on determining duplicate benchmarks. When there are two blocks with same benchmark name in experiment file, we will append them to benchmark list first. After that, we will rename the benchmark if it has test_args: 'story-filter' or 'story-tag-filter'. The new benchmark name will be passed to experiment so that we will treat them as two different runs and get separate tables on the report. BUG=chromium:902466 TEST=Passed all up-to-date unittests, tested and verified with one experiment file containing following benchmark settings: benchmark: webrtc { iterations: 3 test_args: --story-filter=datachannel suite: telemetry_Crosperf } benchmark: webrtc { iterations: 3 test_args: --story-tag-filter=smoothness suite: telemetry_Crosperf } Change-Id: Id733273a5f9f43d149407055c9c0da3b761ddeef Reviewed-on: https://chromium-review.googlesource.com/1321415 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-11-05crosperf: move all_crosbolt_perf benchmark to telemetry_crosperf suiteZhizhou Yang
This patch fixes the issue that all_crosbolt_perf cannot run correctly with crosperf. The construction of this benchmark should sit in telemetry suite. BUG=chromium:901568 TEST=ran all_crosbolt_perf benchmark with crosperf on local DUT Change-Id: I94018cd7ad8348238085270b92c5a7e44ae4bbaa Reviewed-on: https://chromium-review.googlesource.com/1316708 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-07-06crosperf: remove smoothness.tough_webgl_casesTing-Yuan Huang
It is no longer available. TEST=none BUG=none Change-Id: I25437887ec2bd608ba90430a4753e4010ab8a2f5 Reviewed-on: https://chromium-review.googlesource.com/1128364 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@chromium.org>
2018-06-02crosperf: add speedometer2 to toolchain perf tests.Rahul Chaudhry
BUG=None TEST=None Change-Id: I73a8893e892b2941ba1170b1606e4a17744265c5 Reviewed-on: https://chromium-review.googlesource.com/1082954 Commit-Ready: Rahul Chaudhry <rahulchaudhry@chromium.org> Tested-by: Rahul Chaudhry <rahulchaudhry@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2018-04-03crosperf: remove three benchmarks from test suiteYunlian Jiang
Three benchamrks were deleted in upstream, we need to remove them from our test suite. BUG=chromium:828463 TEST=uniitest passes. Change-Id: I5c92ca86217dc4bac79a75cba82dab4a793e0b3a Reviewed-on: https://chromium-review.googlesource.com/992879 Commit-Ready: Yunlian Jiang <yunlian@chromium.org> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-03-08Replacing top_25_smooth with cros_ui_smoothness in crosperfKuo-Hsin Yang
top_25_smooth was replaced in CL:903246. BUG=chromium:818100 TEST=None Change-Id: I5bfe5659b06192237b0a985abb03b99139928e17 Reviewed-on: https://chromium-review.googlesource.com/954809 Commit-Ready: Vovo Yang <vovoy@chromium.org> Tested-by: Vovo Yang <vovoy@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Chung-yih Wang <cywang@google.com>
2017-11-30Remove droameo benchmarks from predefined sets.Luis A. Lozano
They were renamed (grouped) in chromium:777664 but, instead of using the new name, we are going to just remove them from the set. CWP data indicates these benchmarks are not representative. BUG=None TEST=None Change-Id: I034961d4f98eda0cb6ab7e706ac7309afbf23f69 Reviewed-on: https://chromium-review.googlesource.com/802714 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Ting-Yuan Huang <laszio@chromium.org>
2017-10-31Remove startup.cold.blank_page as Chrome perf team removed it.Dean Liao
startup.cold.blank_page was removed by Chrome perf team: https://chromium-review.googlesource.com/c/chromium/src/+/644107 https://bugs.chromium.org/p/chromium/issues/detail?id=575762&desc=2 BUG=chromium:779484 TEST=./run_tests.sh Change-Id: Ib4cd3c2eaed2f496b8e25ab286919895adaa7571 Reviewed-on: https://chromium-review.googlesource.com/742778 Commit-Ready: Shuo-Peng Liao <deanliao@chromium.org> Tested-by: Shuo-Peng Liao <deanliao@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-07-27crosperf: remove duplicated benchmark in all_crosbolt_perfYunlian Jiang
This removes the duplicated benchmark video_PlaybackPerf.h264 in all_crosbolt_perf BUG=None TEST=the duplicated test was removed. Change-Id: Id4822fc1b82986cba2847b2e32ef825e90969c58 Reviewed-on: https://chromium-review.googlesource.com/590359 Commit-Ready: Yunlian Jiang <yunlian@chromium.org> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-04-08[toolchain-utils] Fix remaining lint errors in toolchain-utils.Caroline Tice
In addition to fixing the lint errors, this also fixes the Python formatting issues (ran tc_pyformat on nearly all the files). BUG=chromium:570450 TEST=Ran all crosperf & bisect tool unit tests. Ran afe_lock_machine.py (check machine status) Ran full crosperf test (octane, speedometer, BootPerf) on alex. Change-Id: Ic86f9192801ac67769f3de30f1c5f0d203ce0831 Reviewed-on: https://chromium-review.googlesource.com/471886 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-01-10crosperf: disable cheets benchmarks in crosboltTing-Yuan Huang
They have nothing to do with cros toolchain. TEST=none BUG=none Change-Id: Ie06ccd1cddf7233c55b52fd77e1329311fcff9d9 Reviewed-on: https://chromium-review.googlesource.com/426379 Trybot-Ready: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@chromium.org> Commit-Queue: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org>
2016-12-09crosperf: Add all_graphics_perf and all_crosbolt_perfTing-Yuan Huang
BUG=none TEST=Run them on samus. Change-Id: I2d66351d80ce87f84924fe3bf55abf903fc54829 Reviewed-on: https://chrome-internal-review.googlesource.com/311196 Commit-Ready: Ting-Yuan Huang <laszio@google.com> Tested-by: Ting-Yuan Huang <laszio@google.com> Reviewed-by: Ting-Yuan Huang <laszio@google.com>
2016-11-07Download autotest files with crosperf and add autotest path to test_thatManoj Gupta
when running non-telemetry tests BUG: chromium:647429 TEST:crosperf unit tests, sample experiment file that runs WebGl_Aquarium Change-Id: I067e350fee36596ce269b954773c39a3868632aa Reviewed-on: https://chrome-internal-review.googlesource.com/302296 Commit-Ready: Manoj Gupta <manojgupta@google.com> Tested-by: Manoj Gupta <manojgupta@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-10-27[toolchain-utils] Update page_cycler in nightly tests.Caroline Tice
Currently page_cycler_v2.typical_25 tries to run too many times, causing some of the nightly builders to take nearly 24 hours to complete (they take about 6 hours without page_cycler). Also, page_cycler_v2.typical_25 cannot run locally on some boards (such as daisy). This CL removes page_cycler_v2.typical_25 from the all_toolchain_perf set, and adds it separately to the nightly tests, with only 2 iterations, no retries, and forcing it to not run locallys on the DUT. BUG=chromium:660087 TEST=Tested the new page cycler settings in a crosperf run. Change-Id: Ia68abce4e8ee29a671e58db13f269e531d31736e Reviewed-on: https://chrome-internal-review.googlesource.com/300855 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-09-26[crosperf] Add new results fields for other page_cycler_v2 tests.Caroline Tice
This Cl adds the default result fields for the other page_cycler_v2 tests (already done for page_cycler_v2.typical_25). It also temporarily removes some tests from the all_pagecyclers set, as they appear to have been removed from telemetry. Will remove these permanently when this is confirmed. BUG=chromium:650307 TEST=Ran crosperf with all_pagecyclers. Change-Id: I9e6c3ae140f66a11d792350b74c8897df74d944c Reviewed-on: https://chrome-internal-review.googlesource.com/289865 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-08-29crosperf: Fix a failing test.George Burgess IV
An earlier commit added extra spacing between remotes in crosperf/default_remotes. This resulted in experiment_factory_unittests.py reporting that there were 5 remotes in some cases (2 of which were ''), since we were splitting on one space. It seems sensible to split on any amount of whitespace, instead. BUG=None TEST=./run_tests.sh passes. Change-Id: I95bd85ba92e5e1f5b30b0273b1b4a6c8853b1995 Reviewed-on: https://chrome-internal-review.googlesource.com/281538 Commit-Ready: George Burgess <gbiv@google.com> Tested-by: George Burgess <gbiv@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-08-16[crosperf] Change 'page_cycler.' to 'page_cycler_v2.' everywhere.Caroline Tice
Telemetry is switching to using page_cyler_v2 and will stop supporting old page_cycler soon. This CL switches everything in toolchain-utils to reference page_cycler_v2. BUG=None TEST=Ran unittests in crosperf with no issues. Change-Id: I0e3ce29bfaf483507a9ba528c6c7c5a8fa00f00e Reviewed-on: https://chrome-internal-review.googlesource.com/271598 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-02-18[crosperf] Remove media.tough_video_cases from nightly tests.Caroline Tice
media.tough_video_cases does not have any good result field to use and it has too high a stddev/mean to be a reliable test. This CL removes it from our nightly test set. BUG=None TEST=None Change-Id: Id80a9dd11de36a1fed01d7735f32e98908c7e1b0 Reviewed-on: https://chrome-internal-review.googlesource.com/248615 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-22Fix cros lint errors.Caroline Tice
Also move deprecated scripts to the 'deprecated' directory. BUG=chromiumos:570464 TEST=tested scripts to make sure they still work. Change-Id: I3442a86d898104591233a0849ea0bafb52ecf1f7 Reviewed-on: https://chrome-internal-review.googlesource.com/244221 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-15Fix issues caused by 'None' being default for List fields.Caroline Tice
BUG=None TEST=Tested in my directory. Change-Id: Ib355808db1923b51455f0a2c1d83bc9139030c5d Reviewed-on: https://chrome-internal-review.googlesource.com/242215 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-14Crosperf: Finish fixing unittest problems.Caroline Tice
This patch finishes fixing the unittest problems. They now all work. BUG=chromium:538397 TEST=./run_tests.sh now completes with no errors. Change-Id: Ie501437d199bfafc8a84cd2a95862550992c2cb8 Reviewed-on: https://chrome-internal-review.googlesource.com/241925 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-10Fix various Mock classes & calls so dry runs work.Caroline Tice
The --dry_run option has been failing because parameter changes in real classes have not always been reflected in Mock classes. This CL fixes that. BUG=chromium:545211 TEST=tested both real and --dry_run runs of Crosperf. Change-Id: I0d0aa144e9fb3c40e1a95314691eacffb007cbe4 Reviewed-on: https://chrome-internal-review.googlesource.com/241525 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-09crosperf: fix lint for expeirment_factory.pyYunlian Jiang
BUG=chromium:567921 TEST=the file passes lint checking. Change-Id: I6d2187d11a51f4e87b5bbd8b905e285bb4ed2475 Reviewed-on: https://chrome-internal-review.googlesource.com/241465 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-08Print friendlier error message when no benchmark is given.Luis Lozano
Catch no-benchmarks and no-labels errors earlier to avoid weird error messages later. BUG=chromium:432616 TEST=tested with different experiment files. Change-Id: I8a05dbac4a47a0f005f0b3c175770a0d88cb4a62 Reviewed-on: https://chrome-internal-review.googlesource.com/241096 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-08Re-enable graphics_WebGLAquarium from buildbots testing.Luis Lozano
With some recent changes to the test_that invocation this should work now. Still there is an issue with benchmarks that require compilation for the particular board. That remains to be fixed. BUG=chromium:551710 TEST=None. This is a revert. Change-Id: I547a4853302f11dc58ddc493ee5368b1e2d33c0e Reviewed-on: https://chrome-internal-review.googlesource.com/241035 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-06 10:00:53 +0000