aboutsummaryrefslogtreecommitdiff
path: root/crosperf/benchmark_run_unittest.py
AgeCommit message (Collapse)Author
2020-02-07crosperf: introduce new test_that and skylab run for suite_runnerZhizhou Yang
With client tests wrapped up by crosperf_Wrapper server tests, we need a categorizing for all tests lunched by suite runner. BUG=chromium:984790 TEST=Passed unittest and simple experiment. Change-Id: Ib7dc848cf5c18c3d79db059163ed1217fd7d318f Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1977040 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2020-01-30crosperf: migration to python 3Zhizhou Yang
This patch migrates crosperf and its utils to python 3. TEST=Passed presubmit check; tested with simple experiment locally. BUG=chromium:1011676 Change-Id: Ib2a9f9c7cf6a1bb1d0b42a1dd3d9e3cbb4d70a36 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2003796 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-12-04crosperf: move enable_aslr and turbostat option into dut_configZhizhou Yang
This patch moves enable_aslr and turbostat into dut_config dictionary, since they are all device setup steps, and it make code cleaner and easier to migrate to telemetry_Crosperf. TEST=Passed unittests. BUG=chromium:1020655 Change-Id: I29a649e76591e206efc19bb2b8a8df9aead16575 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1947903 Reviewed-by: Denis Nikitin <denik@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-11-05crosperf: Improve cooldown functionalityDenis Nikitin
"Restart UI" procedure is changed. DUT configuration including cooldown in SuiteRunner is now performed while UI is off. Start UI after DUT configuration is done. The change showed decrease of cooldown waiting time by a factor of 10. Added Total experiment duration time in the report along with total cooldown wait time. Cooldown wait time is now printed per machine (with parallel execution on multiple machines single cooldown time could be misleading). To collect data per machine cooldown_time attribute is moved from SuiteRunner to CrosMachine class. BUG=chromium:966514 TEST=unittests and HW tests on kevin64 passed. Change-Id: Ib3e145cd986ca7242d97b0aeeb7795ba066753fc Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1897236 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-10-16crosperf: Update top stats and cooldown reportDenis Nikitin
Redirect top statistics from benchmark runs into a separate file topstats.log under results_dir directory. Fix "highest 5" usages to show highest usages of a command (instead of a process) per snapshot. Improve mechanism of calculation chrome high CPU load when benchmark is running. Add Cooldown wait time into email report. Fix minor cros lint warnings to unblock repo upload. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I3999efd554cb5a3b27a2ce3fddb2f20714b434fd Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1856818 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-28crosperf: Print top5 commands into logDenis Nikitin
Enable top in nightly tests and print top 5 commands with high CPU usage into log. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I0efd7f0875829c4423e998ec2fbec7436f7ae148 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1829309 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-24crosperf: Fix cros lint warningsDenis Nikitin
Fixed python lint warnings in crosperf scripts. BUG=chromium:993035 TEST=`repo upload` works w/o --no-verify Change-Id: I3dd7b0ea956458e57e8bd77c19a5b8cb13e1d350 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1821449 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-16crosperf: Setup CPU frequencyDenis Nikitin
Added "cpu_freq_pct" in global settings specifying percent of the max frequency which will be set on DUT during benchmark run. The actual value will be chosen from available frequencies less than or equal to the calculated number. Feature requires scaling_available_frequencies. On the platforms not supporting scaling_available_frequencies "cpu_freq_pct" will be ignored. BUG=chromium:966514 TEST=Unittest and local HW tests on veyron, scarlet and kevin64 passed. Change-Id: I175e2836ecf02fcf6b93d196926c7801d7f016c0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1803780 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-09-10crosperf: Add dut config arguments to experimentDenis Nikitin
In global settings added optional arguments: "cooldown_time" - wait time prior running a benchmark (default: 0), "cooldown_temp" - temperature threshold for waiting (default: 40), "governor" - CPU governor (default: performance), "cpu_usage" - Restrict CPU usage to specific configurations (default: all). "turbostat" argument is moved from benchmark to global settings. Current CL does not apply configurations. Instead it just propagates parameters to SuiteRunner class through "dut_config" dictionary. BUG=chromium:966514 TEST=Unitest and local HW tests passed. Change-Id: I1b2a65883e5176fdde49c9858ebe62a097df89cb Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1778515 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-07-22crosperf: introduce skylab field and pass to experiment and labelZhizhou Yang
This is the initial patch to migrate crosperf from autotest to skylab for lab runs. This patch introduces skylab field in experiment, parsed it in settings_factory and restructured label class and experiment class so that further steps can use skylab field. TEST=Passed all unit tests BUG=chromium:984790 Change-Id: I1104fea739977c522938790012f4f3f522f9d65f Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1705234 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-04-18crosperf: generate perf report with correct debug filesZhizhou Yang
This patch fixes the issue in chromium:946588. This patch makes perf report no longer use hard code debug directories. There are several different situations: 1) When running tests on a downloaded image, it will download debug.tgz from gs, extract it to debug_files in /tmp. Options --symfs and --vmlinux will depend on this directory, and throw a warning to user that --kallsyms cannot be applied. 2) If running with downloaded image and debug.tgz could not work, then we will try to use local build, but give user a warning that it may not match real symbols well. 3) When running tests with local build, try to find debug info from /build/$board directory. Thus, this patch added a new field in label, called 'debug_path', if this is manually set in experiment file, then crosperf will directly use the location. Downloading of debug.tgz will only happen when perf_args is set in global settings. TEST=Passed all unit tests, tested with eve and sand. BUG=chromium:946588 Change-Id: I7f35d1216d912c8526d5501748f951face1273aa Reviewed-on: https://chromium-review.googlesource.com/1561780 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-03-14crosperf: Add unittest for disable ASLR CLTiancong Wang
In crrev.com/c/1398522, we disabled ASLR in crosperf by default. In that CL, there's no unittest support and other crosperf unittest might fail because of the CL. This patch is to fix the errors in unittest and also add unittests for the newly added functionalities. BUG=chromium:930332 TEST=Unittests pass Change-Id: I332604245ea2ee1100a498ce021530572e7609ba Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1497036 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Tiancong Wang <tcwang@google.com>
2018-11-16crosperf: Passing dso and weight to result_cache.pyZhizhou Yang
To generate and collect dso sample count, this patch passes dso and weight into result_cache.py, by setting them as members of Benchmark object. Meantime, fixed all related unit tests. BUG=chromium:902785 TEST=Passed all unit tests and sample tests locally. Change-Id: I483a00d199d48fd5e77d882d7ac81bf93b6940df Reviewed-on: https://chromium-review.googlesource.com/1338344 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-04-08[toolchain-utils] Fix remaining lint errors in toolchain-utils.Caroline Tice
In addition to fixing the lint errors, this also fixes the Python formatting issues (ran tc_pyformat on nearly all the files). BUG=chromium:570450 TEST=Ran all crosperf & bisect tool unit tests. Ran afe_lock_machine.py (check machine status) Ran full crosperf test (octane, speedometer, BootPerf) on alex. Change-Id: Ic86f9192801ac67769f3de30f1c5f0d203ce0831 Reviewed-on: https://chromium-review.googlesource.com/471886 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2016-11-07Download autotest files with crosperf and add autotest path to test_thatManoj Gupta
when running non-telemetry tests BUG: chromium:647429 TEST:crosperf unit tests, sample experiment file that runs WebGl_Aquarium Change-Id: I067e350fee36596ce269b954773c39a3868632aa Reviewed-on: https://chrome-internal-review.googlesource.com/302296 Commit-Ready: Manoj Gupta <manojgupta@google.com> Tested-by: Manoj Gupta <manojgupta@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-11-01Update formatting of files before committing changes for autotest. No ↵Manoj Gupta
functional change. BUG: chromium:647429 TEST: crosperf unit tests Change-Id: I827f9ec4d00a0d8c1ae8a9930d7832289e694dc8 Reviewed-on: https://chrome-internal-review.googlesource.com/301457 Commit-Ready: Manoj Gupta <manojgupta@google.com> Tested-by: Manoj Gupta <manojgupta@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-09-13crosperf+cros_utils: Swap to #!/usr/bin/env.George Burgess IV
`cros lint` was recently updated to complain about using #!/usr/bin/python{,2,3} directly. Instead, it prefers /usr/bin/env python{2,3} now. BUG=None TEST=./run_tests.sh passes; linter seems much happier with crosperf. Change-Id: I9b88a7af1f8e03b7c870781c3d8dc1b872a3dfc3 Reviewed-on: https://chrome-internal-review.googlesource.com/286341 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-07-29toolchain-utils: fix benchmark_run_unittest.pyRahul Chaudhry
The 'show_all' argument for CreateFromRun was removed in https://chrome-internal-review.googlesource.com/#/c/255303 BUG=None TEST='./benchmark_run_unittest.py passes' Change-Id: I984393d4db5bf98f0f273c25e366bb31a3e9b845 Reviewed-on: https://chrome-internal-review.googlesource.com/272167 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-07-27[crosperf] Clean up exceptions.Caroline Tice
Found all instances of 'raise Exception' and changed them to raise something slightly more specific. Changed a few 'except Exception' statements appropriately as well. BUG=chromium:566255 TEST=None Change-Id: If2666bd55838342bc71431e5e6ab2157c2c9e125 Reviewed-on: https://chrome-internal-review.googlesource.com/270731 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-02-02Fix lint for toolchain-utils.Han Shen
Tested: run_tests.sh && cros lint on all the modified files. Change-Id: Ife1e9620535f70849c58e932637b7b0f150b13ac Reviewed-on: https://chrome-internal-review.googlesource.com/246650 Commit-Ready: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-26crosperf: Make things work without access to file locks dirDavid Sharp
Before this change, running with --dry_run would result in an error accessing the lock file directory if the user does not have access to the directory. Also disable using file locks in machine_manager_unittest TEST=run crosperf with --dry_run; unit tests BUG=none Change-Id: I8283bc38fd388dfa90fb772e76370a674e86b86f Reviewed-on: https://chrome-internal-review.googlesource.com/245899 Commit-Ready: David Sharp <dhsharp@google.com> Tested-by: David Sharp <dhsharp@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-14Crosperf: Finish fixing unittest problems.Caroline Tice
This patch finishes fixing the unittest problems. They now all work. BUG=chromium:538397 TEST=./run_tests.sh now completes with no errors. Change-Id: Ie501437d199bfafc8a84cd2a95862550992c2cb8 Reviewed-on: https://chrome-internal-review.googlesource.com/241925 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-11Crosperf: Fix many broken unittests.Caroline Tice
Apparently people have been making changes and not running the unittests; many of the unittests were suffering from bit-rot. This CL fixes most of the broken unittests (there are still 3 large ones being worked on -- that will be in another CL). BUG=chromium:567883 TEST=Verified that these unittests work now. Change-Id: I0252a276a035894e70e04c61339cdba148c9bbfd Reviewed-on: https://chrome-internal-review.googlesource.com/241513 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-05-22Remove run_remote_tests references.cmtice
run_remote_tests.sh is about to be removed from ChromeOS. This CL removes all references to run_remote_tests from toolchain-utils. It also updates unittests to make sure they work with recent changes. BUG=crbug:498778 TEST=Verifed results_cache_unittest (and all the other unittests) works with changes. Change-Id: Ic9413bf77a1679e1267b81b182fa7f64da2629b5 Reviewed-on: https://chrome-internal-review.googlesource.com/216884 Reviewed-by: David Sharp <dhsharp@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-10-22Fix issue where crosperf can't find local cros_scratch.cmtice
Crosperf was assuming the user always has an NFS home directory, and was looking for the cros_scratch directory there. This makes crosperf use the user's home directory, whether it's NFS or local. Also change --share_users, which also assumed NFS home directories, to --share_cache, which requires specification of the cache path(s) to use. BUG=None TEST=Tested this with my new local homedir. Change-Id: I28d15467dfbd25b8b237cefa2979d3fd9464a7a7 Reviewed-on: https://chrome-internal-review.googlesource.com/179978 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-06-02Add --cache_only global setting to crosperf.cmtice
The CL adds a new setting, --cache_only, which causes crosperf to ONLY report test results that it finds in the cache (it will NOT rerun any tests, even if no cache results are found). This is partly for implementing a weekly summary report, where we want to compare the nightly results from the past week, but not actually rerun anything. BUG=None TEST=I tested this option while generating weekly test reports, and it worked as it is supposed to. Change-Id: Icb1f5343f76ce999bebd0810bb86ab64a7577313 Reviewed-on: https://chrome-internal-review.googlesource.com/164890 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-05-14Update & complete benchmark_run_unittest.pycmtice
Add tests for all the methods in BenchmarkRun. BUG=Non TEST=Ran unittest. Change-Id: I9046f010ce0e48ba8d897585fec2f95732df9cd3 Reviewed-on: https://chrome-internal-review.googlesource.com/162988 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-21Clean up Crosperf options and help.cmtice
Crosperf's help was completely out of date and incorrect. Fixed that. Also, I cleaned up the options, removed some that were no longer appropriate or not ever used, and made some options global-only. Also removed all references to pyauto. We no longer use or run pyauto tests. Option changes: Completely removed outlier_range (it was not actually doing anything), key_results_only (applied only to pyauto tests), use_test_that (not needed now that we're not using pyauto) and md5sum (never used). Made the following options global-only: rm_chroot_tmp, perf_args, logging_level, show_all_results, board. Removed 'build' from global settings and made it an image-only setting (it made no sense as a global setting). I also adjusted the unittests to work with these changes. BUG=None TEST=I tested all the options in all their various positions; I tested with and wtihout caching; and I tested local and official images. I also ran the unittests. Everything works. Change-Id: I7653663f65c0a3363a1904c0220f1066e72c7f9d Reviewed-on: https://chrome-internal-review.googlesource.com/160758 Reviewed-by: Bhaskar Janakiraman <bjanakiraman@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-11Update unittests to all pass.cmtice
Fix the parameters to the various unittests so they match recent changes and the unittests all pass again. BUG=None TEST=I ran all the unittests with the changes. Change-Id: I083b5127a2ade8f1dbaf2bb173d82183871cb7c7 Reviewed-on: https://chrome-internal-review.googlesource.com/159915 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-10-01crosperf: fix unittest for crosperf.Yunlian Jiang
This fixes several issues for unittesting. Now unittest seems to be working. BUG=None TEST=crosperf/run_tests.sh passes. Change-Id: I1dcef7ba55d45fbd00e453f749a8b10dea0332e6 Reviewed-on: https://chrome-internal-review.googlesource.com/145635 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com>
2013-06-06Adding support of telemetry to crosperfYunlian Jiang
BUG=None TEST=run crosperf with pyauto/telemetry test with/without cache. all pass. Change-Id: If07ac020a9107a79d5780a58fd6dcc924d07f07f Reviewed-on: https://gerrit-int.chromium.org/36594 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-on: https://gerrit-int.chromium.org/39241 Reviewed-by: Yunlian Jiang <yunlian@google.com>
2013-03-15Cleaned up directory after copy of tools from perforce directoryLuis Lozano
Got rid of stale copies of some tools like "crosperf" and moved all files under v14 directory (that came from perforce) into the top directory. BUG=None TEST=None Change-Id: I408d17a36ceb00e74db71403d2351fd466a14f8e Reviewed-on: https://gerrit-int.chromium.org/33887 Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org>
2012-12-20Synced repos to: 64740Ahmad Sharif
2012-05-23Synced repos to: 60708Ahmad Sharif
2012-02-02Synced repos to: 58208Ahmad Sharif
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-18 05:55:04 +0000