aboutsummaryrefslogtreecommitdiff
path: root/crosperf/experiment_runner_unittest.py
AgeCommit message (Collapse)Author
2020-06-05crosperf: Include PID in topstatsDenis Nikitin
Top statistics was showing commands which could combined multiple processes. To include PID in topstats we need to split the commands into separate processes. The process PID is appended to the command name in topstats. The top chrome process is the renderer process running the benchmark. The list size depends on the number of non-chrome processes which we limit to 5. For example with 10 chrome processes in the top with 10 following non-chrome processes the list will show 15 entries. BUG=None TEST=Tested on eve, bob and cheza. Change-Id: Ibf1e61c8cb522aba13cd51a590bb7e24597f66a6 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2227626 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2020-05-26crosperf: introduce option "compress_results" for result directoryzhizhouy
Perf data and report file can occupy large disk space when we copy and store them in the crosperf results directory. This patch introduces a new field for crosperf so that we create a tarball for it to save space. By default compress_result is set to True. BUG=chromium:1079048 TEST=Passed unittest, tested with simple examples. Change-Id: I70db6f3f70dc33bacfccd8833a4327c494b4d6b5 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2212432 Reviewed-by: George Burgess <gbiv@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2020-03-26crosperf: raise error at exit when benchmarks fail to runzhizhouy
Crosperf always returns 0 no matter benchmarks fail or experiment interrupted in the middle. Thus we cannot tell if a run succeeded or not with the exit status of it and it makes our nightly test failures hard to find. In this patch, I changed the behavior of crosperf return value: 1) Crosperf will not generate any report or send email if terminated or all benchmarks fail. It raises RuntimeError stating all benchmarks failing in the end. 2) Crosperf will generate report if benchmarks (but not all) fail, and will raise RuntimeError stating benchmarks partially failing. 3) Crosperf will also copy results json files to local results directory for further info. BUG=chromium:1063703 TEST=Passed all unittests, tested with different failure situations. Change-Id: I998bad51cd7301b9451645d22e8734963bc01aed Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2119231 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2020-01-30crosperf: migration to python 3Zhizhou Yang
This patch migrates crosperf and its utils to python 3. TEST=Passed presubmit check; tested with simple experiment locally. BUG=chromium:1011676 Change-Id: Ib2a9f9c7cf6a1bb1d0b42a1dd3d9e3cbb4d70a36 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2003796 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-11-13crosperf: fix unittest error without prodaccessZhizhou Yang
After locking mechanism changed, crosperf checks if locks_dir is set or not and by default uses X20 location. This requires prodaccess thus unittests will fail without prodaccess if locks_dir is not explicitly set. This patch set all experiment files in unittests to use /tmp for locks_dir. Also fixed some lint error for deprecated functions. TEST=Passed all unittest. BUG=None Change-Id: Iff4623aad9632842010e5efb85a1e75a38bcf73e Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1912767 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-10-16crosperf: Update top stats and cooldown reportDenis Nikitin
Redirect top statistics from benchmark runs into a separate file topstats.log under results_dir directory. Fix "highest 5" usages to show highest usages of a command (instead of a process) per snapshot. Improve mechanism of calculation chrome high CPU load when benchmark is running. Add Cooldown wait time into email report. Fix minor cros lint warnings to unblock repo upload. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I3999efd554cb5a3b27a2ce3fddb2f20714b434fd Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1856818 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-24crosperf: Fix cros lint warningsDenis Nikitin
Fixed python lint warnings in crosperf scripts. BUG=chromium:993035 TEST=`repo upload` works w/o --no-verify Change-Id: I3dd7b0ea956458e57e8bd77c19a5b8cb13e1d350 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1821449 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-13crosperf: Fix unittest execution timeDenis Nikitin
Added mock patches for time.time() and time.sleep() in experiment_runner_unittest.py. Unittest execution time dropped from 12s to <1s. BUG=chromium:989571 TEST=cd <path-to-crosperf>; ./run_tests.sh Change-Id: I60af5305cd022b9a5854c444cfa9c50cbe0a22c0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1801423 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2017-04-08[toolchain-utils] Fix remaining lint errors in toolchain-utils.Caroline Tice
In addition to fixing the lint errors, this also fixes the Python formatting issues (ran tc_pyformat on nearly all the files). BUG=chromium:570450 TEST=Ran all crosperf & bisect tool unit tests. Ran afe_lock_machine.py (check machine status) Ran full crosperf test (octane, speedometer, BootPerf) on alex. Change-Id: Ic86f9192801ac67769f3de30f1c5f0d203ce0831 Reviewed-on: https://chromium-review.googlesource.com/471886 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2016-09-13crosperf+cros_utils: Swap to #!/usr/bin/env.George Burgess IV
`cros lint` was recently updated to complain about using #!/usr/bin/python{,2,3} directly. Instead, it prefers /usr/bin/env python{2,3} now. BUG=None TEST=./run_tests.sh passes; linter seems much happier with crosperf. Change-Id: I9b88a7af1f8e03b7c870781c3d8dc1b872a3dfc3 Reviewed-on: https://chrome-internal-review.googlesource.com/286341 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-09-08crosperf: Make results_report more general.George Burgess IV
The goal of this patch is to allow us to use results_report to generate non-ChromeOS-specific results reports. It's meant to be a nop for regular crosperf runs (except that, in HTML report generation, whitespace in the HTML itself will look a bit different.) Moreover, results_report used to shuffle Experiments around, each of which, unsurprisingly, contained *tons* of data about an experiment. So, part of this patch was reducing results_report's reliance on Experiments, in favor of a new type, BenchmarkResult. Some parts of results_report still rely on Experiments, but only to provide *extra* data. The minimum amount of data needed to make a results report is contained in a BenchmarkResult, and there's a convenient API we can use to make a BenchmarkResult out of an Experiment. This patch also does a massive refactor of results_report, because lots of the code was mildly more icky than it is with this patch applied. The refactor-for-prettiness and refactor-for-BenchmarkResults kind of go hand-in-hand, so it's really hard to split them out. The only part that's not so difficult to split out is the refactor to results_report_templates, but the value of splitting that out is questionable (to me). Speaking of which, all HTML magicks that were in results_report is now in results_report_templates, and now use *actual* Templates. It -- and HTMLReportGenerator -- are hopefully more readable as a result. Next, this makes JSONRsultsReport's GetReport() return a string containing JSON data, rather than calling a callback to write that string to a file. Finally, this includes a change to perf_table. Namely, its removal. It was otherwise unused, and I couldn't get it to even work, so I cleaned it up, made a perf report parser (because I couldn't get the cros_utils one to work, either...), and made sure it functions. If we're able to produce perf reports that cros_utils can parse, I'm happy to back my new parser out; I was primarily using it so I could be sure I didn't break HTML report generation. BUG=chromium:641098 TEST=./run_tests.sh passes Change-Id: I437de9eb39e00c9dd5c223ecd27feaaab544a6fd Reviewed-on: https://chrome-internal-review.googlesource.com/282217 Commit-Ready: George Burgess <gbiv@google.com> Tested-by: George Burgess <gbiv@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-02-13toolchain-utils: disable arguments-differ lint warnings.Rahul Chaudhry
These are ok for overriding the setUp method of unittest.Testcase when using the mock package. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: Id501fe845d4b2cdf9029fc57b20a8e1b26ab7c27 Reviewed-on: https://chrome-internal-review.googlesource.com/248231 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-02-01toolchain-utils: fix attribute-defined-outside-init lint errors.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: If8c79fa74f30c40a67a9f0dee855a38d3e3a9c46 Reviewed-on: https://chrome-internal-review.googlesource.com/246491 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-23toolchain-utils: disable protected-access lint warnings in tests.Rahul Chaudhry
crosperf/experiment_runner_unittest.py and crosperf/machine_manager_unittest.py had a combined total of 53 protected-access lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: Icc31a63f3383068620314fd440460ab82df463a5 Reviewed-on: https://chrome-internal-review.googlesource.com/244868 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-23toolchain-utils: fix class-missing-docstring lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I0640cab2c8d15380badbfbc0db39e3063cc85471 Reviewed-on: https://chrome-internal-review.googlesource.com/244867 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-23toolchain-utils: fix module-missing-docstring lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: Ifda134bf5f26db04551b915c189814b86e750df3 Reviewed-on: https://chrome-internal-review.googlesource.com/244866 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-04toolchain-utils: fix unused-argument lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: Iae6add81a1fb1c929a09808b5870d2a1762ef291 Reviewed-on: https://chrome-internal-review.googlesource.com/243413 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-04toolchain-utils: fix missing-print-function lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I2e0cf0d903b0ca7662132528ee6860ea8eaee83b Reviewed-on: https://chrome-internal-review.googlesource.com/243412 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-04toolchain-utils: apply pyformat to the source files.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I0cb94c77c26941fafb97cca2b9e9bb12cf4c210e Reviewed-on: https://chrome-internal-review.googlesource.com/243411 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-28toolchain-utils: fix shebang related lint warnings.Rahul Chaudhry
This CL fixes all missing-shebang and bad-shebang lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I41bd3516496b246e4e4a8ccc9f97fb25007738a7 Reviewed-on: https://chrome-internal-review.googlesource.com/243136 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-28toolchain-utils: remove unused imports.Rahul Chaudhry
This CL fixes all unused-import lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I145cebe61117388e729cf1b67001cb727fee1b33 Reviewed-on: https://chrome-internal-review.googlesource.com/243108 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-17Fix unittests that recent changes broke.Caroline Tice
BUG=chromium:570522 TEST=Tested in my directory. Change-Id: I197f5f5be3399babc838f4345173625e4d5bd91c Reviewed-on: https://chrome-internal-review.googlesource.com/242565 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-11Crosperf: Fix many broken unittests.Caroline Tice
Apparently people have been making changes and not running the unittests; many of the unittests were suffering from bit-rot. This CL fixes most of the broken unittests (there are still 3 large ones being worked on -- that will be in another CL). BUG=chromium:567883 TEST=Verified that these unittests work now. Change-Id: I0252a276a035894e70e04c61339cdba148c9bbfd Reviewed-on: https://chrome-internal-review.googlesource.com/241513 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-05-22Add email message body to results directory.cmtice
Save the email message body in the results directory. Also update the email message to say where the results directory is. BUG=None TEST=Tested in my chroot. Change-Id: I1da651549db87edc6b781525d443d21f97041d21 Reviewed-on: https://chrome-internal-review.googlesource.com/217265 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-05-20Add unit test for experiment_runner.pycmtice
BUG=None TEST=Ran unittest. Change-Id: I13b2822f57f4201c94215c603a60f12755230f71 Reviewed-on: https://chrome-internal-review.googlesource.com/163833 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-09 11:05:03 +0000