aboutsummaryrefslogtreecommitdiff
path: root/crosperf/benchmark.py
AgeCommit message (Collapse)Author
2019-12-04crosperf: move enable_aslr and turbostat option into dut_configZhizhou Yang
This patch moves enable_aslr and turbostat into dut_config dictionary, since they are all device setup steps, and it make code cleaner and easier to migrate to telemetry_Crosperf. TEST=Passed unittests. BUG=chromium:1020655 Change-Id: I29a649e76591e206efc19bb2b8a8df9aead16575 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1947903 Reviewed-by: Denis Nikitin <denik@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-09-24crosperf: Fix cros lint warningsDenis Nikitin
Fixed python lint warnings in crosperf scripts. BUG=chromium:993035 TEST=`repo upload` works w/o --no-verify Change-Id: I3dd7b0ea956458e57e8bd77c19a5b8cb13e1d350 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1821449 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-08-02crosperf: Add "intel_pstate" and "turbostat" argsDenis Nikitin
In global setting added "intel_pstate" option. Type: string. Values: active, passive, no_hwp, ''(equivalent to active). Default ''. In benchmark setting added "turbostat" option. Type: boolean. Default True. BUG=chromium:966514 TEST=tested on samus, eve, rammus Change-Id: I1533c6d6586aab5acf3665bce68920a0ab3092bc Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1726635 Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: Sean Abraham <seanabraham@chromium.org> Reviewed-by: Sean Abraham <seanabraham@chromium.org> Tested-by: Sean Abraham <seanabraham@chromium.org>
2018-12-20toolchain-utils: Add loading.desktop@@typical to nightly tests.Caroline Tice
page_cycler_v2.typical_25 is deprecated (and flaky!). The recommended replacement is loading.desktop@@typical. This CL updates our nightly performance test infrastructure to handle loading.desktop@@typical. BUG=chromium:916340 TEST=Tested with nightly run on chrotomation2. Change-Id: I6366b3a8e6a2926ad759a209cfded890e3109810 Reviewed-on: https://chromium-review.googlesource.com/1382826 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-11-16crosperf: Passing dso and weight to result_cache.pyZhizhou Yang
To generate and collect dso sample count, this patch passes dso and weight into result_cache.py, by setting them as members of Benchmark object. Meantime, fixed all related unit tests. BUG=chromium:902785 TEST=Passed all unit tests and sample tests locally. Change-Id: I483a00d199d48fd5e77d882d7ac81bf93b6940df Reviewed-on: https://chromium-review.googlesource.com/1338344 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-11-13crosperf: introduce cwp_dso and weight for CWP approximationZhizhou Yang
This patch does initial work to support CWP approximation in crosperf. It introduceds a global option cwp_dso, which is used to specify the type of DSO to use for approximation; an option weight in benchmark setting, which decides the weight of each benchmark in test. This patch also contains some value checks for these two options. BUG=chromium:902785 TEST=Passed all unit tests and sample tests locally. Change-Id: I565c4baf0630ce6c1b62ad0398d05a5f336aac6e Reviewed-on: https://chromium-review.googlesource.com/1323473 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-07-06crosperf: remove smoothness.tough_webgl_casesTing-Yuan Huang
It is no longer available. TEST=none BUG=none Change-Id: I25437887ec2bd608ba90430a4753e4010ab8a2f5 Reviewed-on: https://chromium-review.googlesource.com/1128364 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@chromium.org>
2018-06-05update _estimated_stddev for speedometer2Yunlian Jiang
This updates _estimated_stddev for speedometer2. The value is collected by running speedometer2 on kevin and samus to get the maximum stddev/mean. BUG=chromium:849321 TEST=None Change-Id: If5efa962451a4f221eef4c33a408e88c7d3fee43 Reviewed-on: https://chromium-review.googlesource.com/1087347 Commit-Ready: Yunlian Jiang <yunlian@chromium.org> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-04-08[toolchain-utils] Fix remaining lint errors in toolchain-utils.Caroline Tice
In addition to fixing the lint errors, this also fixes the Python formatting issues (ran tc_pyformat on nearly all the files). BUG=chromium:570450 TEST=Ran all crosperf & bisect tool unit tests. Ran afe_lock_machine.py (check machine status) Ran full crosperf test (octane, speedometer, BootPerf) on alex. Change-Id: Ic86f9192801ac67769f3de30f1c5f0d203ce0831 Reviewed-on: https://chromium-review.googlesource.com/471886 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-03-10crosperf: make default iterations of benchmarks >= 2Ting-Yuan Huang
So as to get standard deviations, which are needed in ttest for p-value. TEST=none BUG=none Change-Id: If59fb46b62c0cb58610507962f8ca13ccc0b7d01 Reviewed-on: https://chromium-review.googlesource.com/452796 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-02-21crosperf: set recommended iterations for benchmarksTing-Yuan Huang
This CL associates estimated standard deviations to each benchmark, according to experiments. The recommended iterations can be specified by setting iterations = 0 in the experiment files. Setting it to numbers greater than 0 will override the default iterations and behaves exactly the same as before. With this change, benchmarks in all_toolchain_perf get no more than 2% margin of error within 90% of time. See crbug.com/673558 for how the standard deviations are estimated. BUG=chromium:673558 TEST=all_toolchain_perf + page_cycler_v2.typical_25 finishes in 3.5 hours for an image on chell. Change-Id: Ie2ed07878c1237ad31a8568ae3fd3fb96cd11f3b Reviewed-on: https://chromium-review.googlesource.com/424915 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2016-07-27[crosperf] Clean up exceptions.Caroline Tice
Found all instances of 'raise Exception' and changed them to raise something slightly more specific. Changed a few 'except Exception' statements appropriately as well. BUG=chromium:566255 TEST=None Change-Id: If2666bd55838342bc71431e5e6ab2157c2c9e125 Reviewed-on: https://chrome-internal-review.googlesource.com/270731 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-02-02Fix lint for toolchain-utils.Han Shen
Tested: run_tests.sh && cros lint on all the modified files. Change-Id: Ife1e9620535f70849c58e932637b7b0f150b13ac Reviewed-on: https://chrome-internal-review.googlesource.com/246650 Commit-Ready: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-07-16Add 'run_local' to run the test harness locally.Ting-Yuan Huang
The option 'run_local: True|False' will be passed into autotest. For example, this is how test_that will be invoked: test_that --arges="run_local=True ..." ... BUG=None TEST=Tested with the folllowing combinations. benchmark: sunspider, smoothness.top_25_smooth perf_args: record -e cycles,instructions / (none) Change-Id: I7315027a7c9433d17a9f3fe54d7e8c3f480ea4f4 Reviewed-on: https://chrome-internal-review.googlesource.com/217370 Reviewed-by: Caroline Tice <cmtice@google.com> Tested-by: Ting-Yuan Huang <laszio@google.com> Commit-Queue: Ting-Yuan Huang <laszio@google.com>
2015-05-28Added benchmark retry support.Luis Lozano
We often get spurious failures from the benchmarks runs. This CL adds support for a "retries" field that will automatically retry the benchmark run by the amount specified. BUG=None TEST=Test by hand. Change-Id: Ib881c40d766e67a7daf855ce9137d3e3d631f12e Reviewed-on: https://chrome-internal-review.googlesource.com/217525 Reviewed-by: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org>
2014-04-29Fix small bugs with non-autotest Telemetry; other cleanups.cmtice
This CL fixes various small problems found during recent testing: - Force show_all_results to be true for 'pure' telemetry case (the results filtering needs the autotest results to work properly) - Fix bug that was inserting extra 'src' into Chrome src path sometimes - Since non-autotest Telemetry and non-Telemetry-Crosperf tests can't handle 'perf' make it a Fatal error to call them with perf args. - Add missing field to example experiment file. - Add status output messages to downlaod_images.py for non-verbose logging. BUG=None TEST=Tested all these fixes on multiple test runs. Change-Id: If8209356c695dc8b21f8627399a1cbd4a858df23 Reviewed-on: https://chrome-internal-review.googlesource.com/161759 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-21Clean up Crosperf options and help.cmtice
Crosperf's help was completely out of date and incorrect. Fixed that. Also, I cleaned up the options, removed some that were no longer appropriate or not ever used, and made some options global-only. Also removed all references to pyauto. We no longer use or run pyauto tests. Option changes: Completely removed outlier_range (it was not actually doing anything), key_results_only (applied only to pyauto tests), use_test_that (not needed now that we're not using pyauto) and md5sum (never used). Made the following options global-only: rm_chroot_tmp, perf_args, logging_level, show_all_results, board. Removed 'build' from global settings and made it an image-only setting (it made no sense as a global setting). I also adjusted the unittests to work with these changes. BUG=None TEST=I tested all the options in all their various positions; I tested with and wtihout caching; and I tested local and official images. I also ran the unittests. Everything works. Change-Id: I7653663f65c0a3363a1904c0220f1066e72c7f9d Reviewed-on: https://chrome-internal-review.googlesource.com/160758 Reviewed-by: Bhaskar Janakiraman <bjanakiraman@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-12-19Change to use test_that by default instead of run_remote_tests.sh.cmtice
BUG=None TEST=Tested this with BootPerServer in my directory. Checked the logs to make sure the correct script was being used (test_that by default now). Change-Id: I39b2901c2ef3c8070c76b15b929df5da6b168c9c Reviewed-on: https://chrome-internal-review.googlesource.com/150695 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-12-10Use default results for Telemetry tests; add option to use all results.cmtice
When running telemetry_Crosperf, use the json file of report defaults to determine which results to actually include in the report. This CL also adds a flag to Crospref, --show_all_results=True, that allows the user to override the defaults and generate a report showing all the values. BUG=None TEST=Tested this both with and without the --show_all_results flag. It works as expected for me. Change-Id: I91364572ebdf3d54bb8b408fbd5a2580923f6829 Reviewed-on: https://chrome-internal-review.googlesource.com/149457 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-08-22Make test_that available for running tests.Caroline Tice
Add a new flag, --use_test_that=True/False, to crosperf, to allow users to tell crosperf to use the test_that script rather than the run_remote_tests.sh script for running their tests. BUG=None TEST=Tested with and without the flag, to verify that it does the right thing. Change-Id: I1a040d23adbf364b4a47320d9fd759a2d46e6689 Reviewed-on: https://gerrit-int.chromium.org/43109 Reviewed-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-06-06Adding support of telemetry to crosperfYunlian Jiang
BUG=None TEST=run crosperf with pyauto/telemetry test with/without cache. all pass. Change-Id: If07ac020a9107a79d5780a58fd6dcc924d07f07f Reviewed-on: https://gerrit-int.chromium.org/36594 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-on: https://gerrit-int.chromium.org/39241 Reviewed-by: Yunlian Jiang <yunlian@google.com>
2013-03-15Cleaned up directory after copy of tools from perforce directoryLuis Lozano
Got rid of stale copies of some tools like "crosperf" and moved all files under v14 directory (that came from perforce) into the top directory. BUG=None TEST=None Change-Id: I408d17a36ceb00e74db71403d2351fd466a14f8e Reviewed-on: https://gerrit-int.chromium.org/33887 Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org>
2012-12-20Synced repos to: 64740Ahmad Sharif
2012-10-09Synced repos to: 63271Ahmad Sharif
2012-02-02Synced repos to: 58208Ahmad Sharif
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-03 19:59:09 +0000