aboutsummaryrefslogtreecommitdiff
path: root/crosperf/benchmark_run.py
AgeCommit message (Collapse)Author
2020-02-07crosperf: introduce new test_that and skylab run for suite_runnerZhizhou Yang
With client tests wrapped up by crosperf_Wrapper server tests, we need a categorizing for all tests lunched by suite runner. BUG=chromium:984790 TEST=Passed unittest and simple experiment. Change-Id: Ib7dc848cf5c18c3d79db059163ed1217fd7d318f Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1977040 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2020-01-30crosperf: migration to python 3Zhizhou Yang
This patch migrates crosperf and its utils to python 3. TEST=Passed presubmit check; tested with simple experiment locally. BUG=chromium:1011676 Change-Id: Ib2a9f9c7cf6a1bb1d0b42a1dd3d9e3cbb4d70a36 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2003796 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-12-04crosperf: move enable_aslr and turbostat option into dut_configZhizhou Yang
This patch moves enable_aslr and turbostat into dut_config dictionary, since they are all device setup steps, and it make code cleaner and easier to migrate to telemetry_Crosperf. TEST=Passed unittests. BUG=chromium:1020655 Change-Id: I29a649e76591e206efc19bb2b8a8df9aead16575 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1947903 Reviewed-by: Denis Nikitin <denik@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-11-05crosperf: Improve cooldown functionalityDenis Nikitin
"Restart UI" procedure is changed. DUT configuration including cooldown in SuiteRunner is now performed while UI is off. Start UI after DUT configuration is done. The change showed decrease of cooldown waiting time by a factor of 10. Added Total experiment duration time in the report along with total cooldown wait time. Cooldown wait time is now printed per machine (with parallel execution on multiple machines single cooldown time could be misleading). To collect data per machine cooldown_time attribute is moved from SuiteRunner to CrosMachine class. BUG=chromium:966514 TEST=unittests and HW tests on kevin64 passed. Change-Id: Ib3e145cd986ca7242d97b0aeeb7795ba066753fc Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1897236 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-10-16crosperf: Update top stats and cooldown reportDenis Nikitin
Redirect top statistics from benchmark runs into a separate file topstats.log under results_dir directory. Fix "highest 5" usages to show highest usages of a command (instead of a process) per snapshot. Improve mechanism of calculation chrome high CPU load when benchmark is running. Add Cooldown wait time into email report. Fix minor cros lint warnings to unblock repo upload. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I3999efd554cb5a3b27a2ce3fddb2f20714b434fd Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1856818 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-28crosperf: Print top5 commands into logDenis Nikitin
Enable top in nightly tests and print top 5 commands with high CPU usage into log. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I0efd7f0875829c4423e998ec2fbec7436f7ae148 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1829309 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-24crosperf: Fix cros lint warningsDenis Nikitin
Fixed python lint warnings in crosperf scripts. BUG=chromium:993035 TEST=`repo upload` works w/o --no-verify Change-Id: I3dd7b0ea956458e57e8bd77c19a5b8cb13e1d350 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1821449 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-10crosperf: Add dut config arguments to experimentDenis Nikitin
In global settings added optional arguments: "cooldown_time" - wait time prior running a benchmark (default: 0), "cooldown_temp" - temperature threshold for waiting (default: 40), "governor" - CPU governor (default: performance), "cpu_usage" - Restrict CPU usage to specific configurations (default: all). "turbostat" argument is moved from benchmark to global settings. Current CL does not apply configurations. Instead it just propagates parameters to SuiteRunner class through "dut_config" dictionary. BUG=chromium:966514 TEST=Unitest and local HW tests passed. Change-Id: I1b2a65883e5176fdde49c9858ebe62a097df89cb Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1778515 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-03-14crosperf: Add unittest for disable ASLR CLTiancong Wang
In crrev.com/c/1398522, we disabled ASLR in crosperf by default. In that CL, there's no unittest support and other crosperf unittest might fail because of the CL. This patch is to fix the errors in unittest and also add unittests for the newly added functionalities. BUG=chromium:930332 TEST=Unittests pass Change-Id: I332604245ea2ee1100a498ce021530572e7609ba Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1497036 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Tiancong Wang <tcwang@google.com>
2019-03-06crosperf: enabling result caching for cwp modeZhizhou Yang
This patch provides support for result caching when cwp_dso is specified. This patch introduced cwp_dso into ResultsCache class, so that cwp_dso info will be stored while caching. When populating from cache, samples will be re-calculated with the perf.data file that has been cached before. This patch also: 1) Fixed return value of GetSamples(). 2) Not need to pass weight to result. 3) Added unit tests. BUG=chromium:905854 TEST=Passed all unit tests; Tested with --rerun=True/False, both on cwp and general mode, to make sure cache works. Change-Id: I297fa0aaad781b4a665a4a29bb6fdb4bca0d790b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1501458 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-02-12crosperf: Disable ASLR in crosperf by defaultTiancong Wang
Disabling Address Space Layout Randomization (ASLR) can reduce much variance when measuring performance of benchmark from run to run. This CL will disable ASLR in crosperf by default and add a flag '--enable_aslr' for users to enable it for other testing purpose. BUG=chromium:930332 TEST=Tested locally for several boards and several benchmarks Change-Id: Icfbe477a99ef9a68b503e4979cf2ff47d07a128f Reviewed-on: https://chromium-review.googlesource.com/1398522 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Tiancong Wang <tcwang@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-11-16crosperf: Passing dso and weight to result_cache.pyZhizhou Yang
To generate and collect dso sample count, this patch passes dso and weight into result_cache.py, by setting them as members of Benchmark object. Meantime, fixed all related unit tests. BUG=chromium:902785 TEST=Passed all unit tests and sample tests locally. Change-Id: I483a00d199d48fd5e77d882d7ac81bf93b6940df Reviewed-on: https://chromium-review.googlesource.com/1338344 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-04-08[toolchain-utils] Fix remaining lint errors in toolchain-utils.Caroline Tice
In addition to fixing the lint errors, this also fixes the Python formatting issues (ran tc_pyformat on nearly all the files). BUG=chromium:570450 TEST=Ran all crosperf & bisect tool unit tests. Ran afe_lock_machine.py (check machine status) Ran full crosperf test (octane, speedometer, BootPerf) on alex. Change-Id: Ic86f9192801ac67769f3de30f1c5f0d203ce0831 Reviewed-on: https://chromium-review.googlesource.com/471886 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2016-08-10Random nit fixes in crosperf; NFC.George Burgess IV
This also removes results_sorter.py, which seems to be unused and untested. TEST=./run_tests.sh passes BUG=None Change-Id: I926affb23c78a8a0aedea4ab279da590f8cf93b5 Reviewed-on: https://chrome-internal-review.googlesource.com/273995 Commit-Ready: George Burgess <gbiv@google.com> Tested-by: George Burgess <gbiv@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-07-27[crosperf] Clean up exceptions.Caroline Tice
Found all instances of 'raise Exception' and changed them to raise something slightly more specific. Changed a few 'except Exception' statements appropriately as well. BUG=chromium:566255 TEST=None Change-Id: If2666bd55838342bc71431e5e6ab2157c2c9e125 Reviewed-on: https://chrome-internal-review.googlesource.com/270731 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-04-14[crosperf] Fix more lint errors.Caroline Tice
BUG=chromium:567921 Test=Tested in my directory. Change-Id: Ibe128dd030887e0372380fb8f95a5e37ee63feeb Reviewed-on: https://chrome-internal-review.googlesource.com/255303 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-03-05[crosperf] Fix missing chrome_version problem (v2).Caroline Tice
When generating the json reports, occasionally the chrome_version comes up blank (this can happen when reading some results from the cache, for example). A blank chrome_version messes up our dremel database scripts. This CL updates the read-cache scenario to find the chrome_version in the cache keys (human-readable) file. BUG=chromium:591494 TEST=Tested with experiment that was generating blank Chrome versions and this fixed the problem. Change-Id: I29985581aa00bc30fb65720fb9d197f1475445e4 Reviewed-on: https://chrome-internal-review.googlesource.com/249960 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-02-02Fix lint for toolchain-utils.Han Shen
Tested: run_tests.sh && cros lint on all the modified files. Change-Id: Ife1e9620535f70849c58e932637b7b0f150b13ac Reviewed-on: https://chrome-internal-review.googlesource.com/246650 Commit-Ready: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-16crosperf: clean lint warnings.Yunlian Jiang
BUG=chromium:567921 TEST=these three files are lint warning free. Change-Id: I23f9120283fc82958abcc16f47cd29d5800bd181 Reviewed-on: https://chrome-internal-review.googlesource.com/242485 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-11Crosperf: Fix many broken unittests.Caroline Tice
Apparently people have been making changes and not running the unittests; many of the unittests were suffering from bit-rot. This CL fixes most of the broken unittests (there are still 3 large ones being worked on -- that will be in another CL). BUG=chromium:567883 TEST=Verified that these unittests work now. Change-Id: I0252a276a035894e70e04c61339cdba148c9bbfd Reviewed-on: https://chrome-internal-review.googlesource.com/241513 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-11crosperf: clean lint warnings.Yunlian Jiang
BUG=chromium:567921 TEST=machine_manager.py is clean. Change-Id: Ia3355ee78d15419ca531f358812f7882a64a6cbe Reviewed-on: https://chrome-internal-review.googlesource.com/241491 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-09crosperf: use cros_utils instead of utils.Yunlian Jiang
We got some lint warnings about some module is not in utils. So we change the name. This CL creates a symbolic link cros_utils from utils. Once we are sure that we clean all the code, we can rename the utils to cros_utils directly. BUG=chromium:567921 chromium:568195 TEST=the lint warning is gone. Change-Id: I776e45c7bdd75d1f065ea5797f3bcff457203ca2 Reviewed-on: https://chrome-internal-review.googlesource.com/241457 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-10-14Make sure benchmark_runs check the cache for non-schedv2 paths.Caroline Tice
The recent changes for schedv2 caused the cache to NOT be read/checked if we were not using schedv2. This fixes that problem. BUG=None TEST=Tested in my directory with and without --noschedv2 flag. Change-Id: I638b22554f72cf1ea0fdc4e91a326218febbf250 Reviewed-on: https://chrome-internal-review.googlesource.com/234251 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-09-24Fix cache reading problem.Han Shen
The previous implementation images a machine to match the br(benchmarkrun)'s label before ReadCache is performed, that image step is a huge waste. Now re-implemented by starting a few threads at the very beginning to go through all brs, and put cache-hit brs into _cached_br_list. Then we start the DutWorker thread to go through _cached_br_list, which does not require dut to bear any particular image. After this step, we proceed as usual. Unittest cases also added. Tested by manullay launch a crosperf and examining no re-image is done. Change-Id: Ib611937f9fa28ae9bedce8a1a01ed303f14f838a Reviewed-on: https://chrome-internal-review.googlesource.com/231585 Commit-Ready: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-09-15Get Chrome version and store in cache & json files.Caroline Tice
This updates crosperf to query the DUT for the version of Chrome running on it, and to store the Chrome version both in the json reports and in the human-readable cache keys file. BUG=None TEST=None Change-Id: Ie8215528b8119f95306cad694e6898cc4810730e Reviewed-on: https://chrome-internal-review.googlesource.com/230537 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-09-10Store machine description info in cache and json reports.Caroline Tice
This CL updates the cache mechanism to store the machine spec info (the checksum string) in the cache, as well as storing the machine information in the json reports (to go into the dashboard archive). It also updates the json reports to always include the full set of result fields for a test, and it verifies that the old cache directory exists before trying to remove it. BUG=None TEST=Tested in my directory, with --cache_only=True, --rerun=True and --json_report=True Change-Id: Ib12bae4ca10141de0b972e3a7aeee83c7b59049b Reviewed-on: https://chrome-internal-review.googlesource.com/229765 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-09-07Remove references to NonMatchineMachines exception.Caroline Tice
NonMatchingMachines was removed in a CL yesterday, but a few references to it were missed. This removes those references. BUG=None TEST=None Change-Id: Iadc75d26bce6193d30c23c22b5fc1a529103b607 Reviewed-on: https://chrome-internal-review.googlesource.com/229501 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-09-03Various small cache cleanups.Caroline Tice
Add the machine used for running the test to benchmark run; use the actual data from that machine for generating the cache entry for the run. Also make sure, when calculating the common checksums for looking machines up in the cache, that all the machines being used actually have the same checksum. Also add a cache_keys.txt file to the cache directory, to allow people to see what was used for generating the cache entry/checksums. BUG=None TEST=Tested in my directory. Change-Id: Ia4cf8316e920a24becf5d12fd4c8f9a3e30a180f Reviewed-on: https://chrome-internal-review.googlesource.com/228854 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-08-20Crosperf schedv2 (1) - new option and integrating new scheduler.Han Shen
This Cl introduces a new option '--schedv2' which uses the new scheduler to allocate jobs (benchmark_runs) to duts. With this option, schedv2 takes control of storing/allocating jobs and reimage machines using the new algorithm. This CL leaves actual reimaging and running jobs non-op (a random time sleep is used for each such op, which would be replace in later CLs.) You may try this CL like this and see how schedv2 works - crosperf --locks_dir=/usr/local/google/home/shenhan/tmp --use_file_locks=True --logging_level=verbose --schedv2 some.exp Change-Id: If5bb7751b466c39e54c93fe8f0b4e363be6d9165 Reviewed-on: https://chrome-internal-review.googlesource.com/225515 Commit-Queue: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-07-16Add 'run_local' to run the test harness locally.Ting-Yuan Huang
The option 'run_local: True|False' will be passed into autotest. For example, this is how test_that will be invoked: test_that --arges="run_local=True ..." ... BUG=None TEST=Tested with the folllowing combinations. benchmark: sunspider, smoothness.top_25_smooth perf_args: record -e cycles,instructions / (none) Change-Id: I7315027a7c9433d17a9f3fe54d7e8c3f480ea4f4 Reviewed-on: https://chrome-internal-review.googlesource.com/217370 Reviewed-by: Caroline Tice <cmtice@google.com> Tested-by: Ting-Yuan Huang <laszio@google.com> Commit-Queue: Ting-Yuan Huang <laszio@google.com>
2014-10-22Fix issue where crosperf can't find local cros_scratch.cmtice
Crosperf was assuming the user always has an NFS home directory, and was looking for the cros_scratch directory there. This makes crosperf use the user's home directory, whether it's NFS or local. Also change --share_users, which also assumed NFS home directories, to --share_cache, which requires specification of the cache path(s) to use. BUG=None TEST=Tested this with my new local homedir. Change-Id: I28d15467dfbd25b8b237cefa2979d3fd9464a7a7 Reviewed-on: https://chrome-internal-review.googlesource.com/179978 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-06-02Add --cache_only global setting to crosperf.cmtice
The CL adds a new setting, --cache_only, which causes crosperf to ONLY report test results that it finds in the cache (it will NOT rerun any tests, even if no cache results are found). This is partly for implementing a weekly summary report, where we want to compare the nightly results from the past week, but not actually rerun anything. BUG=None TEST=I tested this option while generating weekly test reports, and it worked as it is supposed to. Change-Id: Icb1f5343f76ce999bebd0810bb86ab64a7577313 Reviewed-on: https://chrome-internal-review.googlesource.com/164890 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-05-21Better handling of "not identical machines" failure.cmtice
Nightly tests, especially on the x86-generic boxes, fail frequently because the DUTs have different board images on them, so Crosperf decides the machines are not identical and refuses to run the tests. With this CL, if the machine_manager finds that the machines fail the "identical" test, it will try to push the same image onto all the machines, and then check them again to see if they are the same. It only tries this once; if they fail the check the second time around, it is still a fatal failure. This should eliminate many of the unnecessary failures in our nightly tests. This CL also fixes a small bug in the auto-delete script (this fix has been running for a while on mobiletc-prebuild, but never got committed). BUG=None TEST=Ran several iterations where I forced the first "identical" check to fail. The changes worked. Change-Id: Ied2a55e5d3e2789e58a503aef03269888954b579 Reviewed-on: https://chrome-internal-review.googlesource.com/163334 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-05-14Update & complete benchmark_run_unittest.pycmtice
Add tests for all the methods in BenchmarkRun. BUG=Non TEST=Ran unittest. Change-Id: I9046f010ce0e48ba8d897585fec2f95732df9cd3 Reviewed-on: https://chrome-internal-review.googlesource.com/162988 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-11Update unittests to all pass.cmtice
Fix the parameters to the various unittests so they match recent changes and the unittests all pass again. BUG=None TEST=I ran all the unittests with the changes. Change-Id: I083b5127a2ade8f1dbaf2bb173d82183871cb7c7 Reviewed-on: https://chrome-internal-review.googlesource.com/159915 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-03-14Add --logging_level option to crosperf.cmtice
Add --logging_level={verbose, average, quiet} option to crosperf, optionally reducing the amount of logging output. "verbose" gives the same output Crosperf always gave; "quiet" gives minimal output, and "average" is in the middle. The default for now is "verbose". Doing this also involved adding logging levels to the command_executer, and to image_chromeos (again, with the default being "verbose"). BUG=None TEST=Tested all three levels of logging, with and without having to reimage the DUT. It all worked properly. Change-Id: Icf8c9a6831fe25202adcb624c6c7e1d1a7ac25a5 Reviewed-on: https://chrome-internal-review.googlesource.com/156883 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-03-06Fix bug introduced with the --show_all_results flag, using cache.cmtice
The call to self._ProcessResults is supposed to take the show_all_results flag as an argument. This was overlooked in the read-it-out-of-the-cache path when this change was first introduced. This CL fixes that. BUG=None TEST=I tested cached and non-cached runs, with and without the --show_all_results option. Everything worked as it should. Change-Id: Ibf8f206406b0a4535d48b598709e62135289f993 Reviewed-on: https://chrome-internal-review.googlesource.com/156367 Reviewed-by: Yunlian Jiang <yunlian@google.com> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com>
2013-12-10Use default results for Telemetry tests; add option to use all results.cmtice
When running telemetry_Crosperf, use the json file of report defaults to determine which results to actually include in the report. This CL also adds a flag to Crospref, --show_all_results=True, that allows the user to override the defaults and generate a report showing all the values. BUG=None TEST=Tested this both with and without the --show_all_results flag. It works as expected for me. Change-Id: I91364572ebdf3d54bb8b408fbd5a2580923f6829 Reviewed-on: https://chrome-internal-review.googlesource.com/149457 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-12-02Add the ability to pass arguments through test_that to Telemetry'scmtice
run_benchmark. BUG=None TEST=I have tested this with my own crosperf, and test_that is passing the arguments along properly. The telemetry_Crosperf autotest also needs to be updated before this completely works. Change-Id: Ic887380b297d966229ac21ba4e7e909e7248fcb0 Reviewed-on: https://chrome-internal-review.googlesource.com/148485 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-10-01crosperf: fix unittest for crosperf.Yunlian Jiang
This fixes several issues for unittesting. Now unittest seems to be working. BUG=None TEST=crosperf/run_tests.sh passes. Change-Id: I1dcef7ba55d45fbd00e453f749a8b10dea0332e6 Reviewed-on: https://chrome-internal-review.googlesource.com/145635 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com>
2013-08-22Make test_that available for running tests.Caroline Tice
Add a new flag, --use_test_that=True/False, to crosperf, to allow users to tell crosperf to use the test_that script rather than the run_remote_tests.sh script for running their tests. BUG=None TEST=Tested with and without the flag, to verify that it does the right thing. Change-Id: I1a040d23adbf364b4a47320d9fd759a2d46e6689 Reviewed-on: https://gerrit-int.chromium.org/43109 Reviewed-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-06-06Adding support of telemetry to crosperfYunlian Jiang
BUG=None TEST=run crosperf with pyauto/telemetry test with/without cache. all pass. Change-Id: If07ac020a9107a79d5780a58fd6dcc924d07f07f Reviewed-on: https://gerrit-int.chromium.org/36594 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-on: https://gerrit-int.chromium.org/39241 Reviewed-by: Yunlian Jiang <yunlian@google.com>
2013-05-23crosperf: remove dead machine in the runtime.Yunlian Jiang
This checks whether a machine is reacheable after a test run. If the machine is dead, we remove it from the machine list in the machine manager. BUG=None TEST=crosperf a simple case passes. Change-Id: Ie40a4a41715f2dc0349666079faa599b3c8aa667 Reviewed-on: https://gerrit-int.chromium.org/38578 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com>
2013-03-15Cleaned up directory after copy of tools from perforce directoryLuis Lozano
Got rid of stale copies of some tools like "crosperf" and moved all files under v14 directory (that came from perforce) into the top directory. BUG=None TEST=None Change-Id: I408d17a36ceb00e74db71403d2351fd466a14f8e Reviewed-on: https://gerrit-int.chromium.org/33887 Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org>
2012-12-20Synced repos to: 64740Ahmad Sharif
2012-10-09Synced repos to: 63271Ahmad Sharif
2012-05-23Synced repos to: 60708Ahmad Sharif
2012-05-18Synced repos to: 60588Ahmad Sharif
2012-05-07Synced repos to: 60208Ahmad Sharif
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-17 20:26:50 +0000