aboutsummaryrefslogtreecommitdiff
path: root/crosperf/results_cache_unittest.py
AgeCommit message (Collapse)Author
2020-09-18crosperf: Add verification of per-process profilesDenis Nikitin
Verification is very helpful when we record per-process perf profile and collect top statistics. Using verification we can make sure that perf profiles are collected from the hottest functions in top statistics. BUG=chromium:1098098 TEST=unittests and crosperf with per-process profiling are passing. Change-Id: Ib79b070babcdea38932ba709ac3e14fa3311f842 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2407223 Reviewed-by: Tiancong Wang <tcwang@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Denis Nikitin <denik@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2020-06-20crosperf: Save cpuinfo and top logsDenis Nikitin
Copy cpuinfo and top logs into results directory. BUG=None TEST=tested on local DUT. Change-Id: Ic868894d681db4a27f78e378945e1e565d4a5cc7 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2255007 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Commit-Queue: Denis Nikitin <denik@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2020-06-05crosperf: Include PID in topstatsDenis Nikitin
Top statistics was showing commands which could combined multiple processes. To include PID in topstats we need to split the commands into separate processes. The process PID is appended to the command name in topstats. The top chrome process is the renderer process running the benchmark. The list size depends on the number of non-chrome processes which we limit to 5. For example with 10 chrome processes in the top with 10 following non-chrome processes the list will show 15 entries. BUG=None TEST=Tested on eve, bob and cheza. Change-Id: Ibf1e61c8cb522aba13cd51a590bb7e24597f66a6 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2227626 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2020-03-26crosperf: raise error at exit when benchmarks fail to runzhizhouy
Crosperf always returns 0 no matter benchmarks fail or experiment interrupted in the middle. Thus we cannot tell if a run succeeded or not with the exit status of it and it makes our nightly test failures hard to find. In this patch, I changed the behavior of crosperf return value: 1) Crosperf will not generate any report or send email if terminated or all benchmarks fail. It raises RuntimeError stating all benchmarks failing in the end. 2) Crosperf will generate report if benchmarks (but not all) fail, and will raise RuntimeError stating benchmarks partially failing. 3) Crosperf will also copy results json files to local results directory for further info. BUG=chromium:1063703 TEST=Passed all unittests, tested with different failure situations. Change-Id: I998bad51cd7301b9451645d22e8734963bc01aed Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2119231 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2020-01-30crosperf: migration to python 3Zhizhou Yang
This patch migrates crosperf and its utils to python 3. TEST=Passed presubmit check; tested with simple experiment locally. BUG=chromium:1011676 Change-Id: Ib2a9f9c7cf6a1bb1d0b42a1dd3d9e3cbb4d70a36 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2003796 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-12-11crosperf: migrate device setup functions for telemetry_CrosperfZhizhou Yang
This is a patch working together with crrev.com/c/1949606 to migrate device setup functions from suite_runner to autotest telmetry_Crosperf. In this patch, we modified the logic of different suite types along with skylab mode for suite_runner. In result_cache, we handle the wait time log file to accumulate cool down time per machine. We also move intel_pstate kernel updating code into SetupDevice() since it doesn't increase overhead by checking every iteration. TEST=Passed all unittests; tested with different benchmark modes (Note that currently skylab server tests cannot be tested because of regression: crbug.com/984103). BUG=chromium:1020655 Cq-Depend: chromium:1949606 Change-Id: If8e5099d096d2568d4f54584a8fcfd3c0f99c8f8 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1949626 Reviewed-by: George Burgess <gbiv@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Commit-Queue: Denis Nikitin <denik@chromium.org>
2019-10-16crosperf: Update top stats and cooldown reportDenis Nikitin
Redirect top statistics from benchmark runs into a separate file topstats.log under results_dir directory. Fix "highest 5" usages to show highest usages of a command (instead of a process) per snapshot. Improve mechanism of calculation chrome high CPU load when benchmark is running. Add Cooldown wait time into email report. Fix minor cros lint warnings to unblock repo upload. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I3999efd554cb5a3b27a2ce3fddb2f20714b434fd Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1856818 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-28crosperf: Print top5 commands into logDenis Nikitin
Enable top in nightly tests and print top 5 commands with high CPU usage into log. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I0efd7f0875829c4423e998ec2fbec7436f7ae148 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1829309 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-26crosperf: mark a run as failed if samples collected is 0Zhizhou Yang
There is situation that in cwp mode, test runs correctly but not samples are collected. For this case, we want to mark the run as failure so that we could have correct speedup calculation and user can notice it. BUG=chromium:927554 TEST=Passed unit tests Change-Id: I4d4f9bd25159e8d3d0b14a84f7f566593e1438ac Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1594288 Tested-by: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
2019-09-24crosperf: Fix cros lint warningsDenis Nikitin
Fixed python lint warnings in crosperf scripts. BUG=chromium:993035 TEST=`repo upload` works w/o --no-verify Change-Id: I3dd7b0ea956458e57e8bd77c19a5b8cb13e1d350 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1821449 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-24crosperf: add per tag story summary when parsing histogramsJian Cai
Add support to collecting results based on story tags in histograms. This allows us to remove the workaround for loading.desktop and deprecate chartjson. BUG=chromium:1000497 TEST=local tests. Change-Id: I8249d00b3d2d5d761c8a3bc24ef2fd95a3e1115b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1783837 Tested-by: Jian Cai <jiancai@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Tiancong Wang <tcwang@google.com>
2019-08-19crosperf: Add ARM CPU stats in crosperf reportDenis Nikitin
Check for cpustats.log file in results_dir and extract data with CPU frequncy and temperature. Calculate avg/min/max values from the measurements and add to result keyvals. The parser attempts to read turbostat.log first (which comes from Intel devices along with cpustats.log) and only if it doesn't exist or empty switches to cpustats.log (usually ARM devices). BUG=chromium:966514 TEST=tested locally on veyron, scarlet, kevin64 (arm), eve (intel) Change-Id: I12991b39a0f1acb6197ca663a6876b0f013d2026 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1753164 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-07-28crosperf: Add cpu statistics into results reportDenis Nikitin
Check for turbostat.log file in results directory and if file exists parse the context and extract measurements of CPU Frequency and temperature for ALL cores (not each). Keys added to keyvals: 'cpufreq_min', 'cpufreq_max', 'cpufreq_avg', 'cputemp'. BUG=chromium:966514 TEST=tested all crosbolt perf benchmarks Change-Id: If90d26664c70ccb73750f17573b89933fdb048f4 Reviewed-on: https://chromium-review.googlesource.com/1709824 Tested-by: Denis Nikitin <denik@chromium.org> Commit-Ready: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-07-22crosperf: introduce skylab field and pass to experiment and labelZhizhou Yang
This is the initial patch to migrate crosperf from autotest to skylab for lab runs. This patch introduces skylab field in experiment, parsed it in settings_factory and restructured label class and experiment class so that further steps can use skylab field. TEST=Passed all unit tests BUG=chromium:984790 Change-Id: I1104fea739977c522938790012f4f3f522f9d65f Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1705234 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-04-18crosperf: generate perf report with correct debug filesZhizhou Yang
This patch fixes the issue in chromium:946588. This patch makes perf report no longer use hard code debug directories. There are several different situations: 1) When running tests on a downloaded image, it will download debug.tgz from gs, extract it to debug_files in /tmp. Options --symfs and --vmlinux will depend on this directory, and throw a warning to user that --kallsyms cannot be applied. 2) If running with downloaded image and debug.tgz could not work, then we will try to use local build, but give user a warning that it may not match real symbols well. 3) When running tests with local build, try to find debug info from /build/$board directory. Thus, this patch added a new field in label, called 'debug_path', if this is manually set in experiment file, then crosperf will directly use the location. Downloading of debug.tgz will only happen when perf_args is set in global settings. TEST=Passed all unit tests, tested with eve and sand. BUG=chromium:946588 Change-Id: I7f35d1216d912c8526d5501748f951face1273aa Reviewed-on: https://chromium-review.googlesource.com/1561780 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-03-07crosperf: do not put key 'samples' in result.keyval when test failedZhizhou Yang
This patch fixed two issues: 1) When determining how many iterations pass/fail, we originally check if samples exists or not in result.keyval. So it always comes up with "all pass" even if test fails and samples equals to 0. 2) Stddev will become large when we include 0s in samples. After this change, the value in failed cell with be 'None', and not be calculated in stddev and avg. BUG=chromium:938582 TEST=Tested with runs with failures, for cwp and general mode. Passed all unittests. Change-Id: If14eff40631be1080f6713f2bfb631843cb2ab62 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1504632 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-03-06crosperf: enabling result caching for cwp modeZhizhou Yang
This patch provides support for result caching when cwp_dso is specified. This patch introduced cwp_dso into ResultsCache class, so that cwp_dso info will be stored while caching. When populating from cache, samples will be re-calculated with the perf.data file that has been cached before. This patch also: 1) Fixed return value of GetSamples(). 2) Not need to pass weight to result. 3) Added unit tests. BUG=chromium:905854 TEST=Passed all unit tests; Tested with --rerun=True/False, both on cwp and general mode, to make sure cache works. Change-Id: I297fa0aaad781b4a665a4a29bb6fdb4bca0d790b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1501458 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2018-11-16crosperf: Passing dso and weight to result_cache.pyZhizhou Yang
To generate and collect dso sample count, this patch passes dso and weight into result_cache.py, by setting them as members of Benchmark object. Meantime, fixed all related unit tests. BUG=chromium:902785 TEST=Passed all unit tests and sample tests locally. Change-Id: I483a00d199d48fd5e77d882d7ac81bf93b6940df Reviewed-on: https://chromium-review.googlesource.com/1338344 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-04-08[toolchain-utils] Fix remaining lint errors in toolchain-utils.Caroline Tice
In addition to fixing the lint errors, this also fixes the Python formatting issues (ran tc_pyformat on nearly all the files). BUG=chromium:570450 TEST=Ran all crosperf & bisect tool unit tests. Ran afe_lock_machine.py (check machine status) Ran full crosperf test (octane, speedometer, BootPerf) on alex. Change-Id: Ic86f9192801ac67769f3de30f1c5f0d203ce0831 Reviewed-on: https://chromium-review.googlesource.com/471886 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-01-23remove perf binary from toolchain-utils.Yunlian Jiang
This reverts this CL https://chrome-internal-review.googlesource.com/#/c/157756/, because the bug in perf is gone. BUG=chromium:665988 TEST=run a crosperf with perf, the report report is correct. No 20-50% of the report attributed to 'chrome [unknown]'. run_test.sh passes. Change-Id: I4b2b18e9653d95bde5caf076114bf53a84c2a4fd Reviewed-on: https://chromium-review.googlesource.com/431234 Commit-Ready: Yunlian Jiang <yunlian@chromium.org> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2016-11-07Download autotest files with crosperf and add autotest path to test_thatManoj Gupta
when running non-telemetry tests BUG: chromium:647429 TEST:crosperf unit tests, sample experiment file that runs WebGl_Aquarium Change-Id: I067e350fee36596ce269b954773c39a3868632aa Reviewed-on: https://chrome-internal-review.googlesource.com/302296 Commit-Ready: Manoj Gupta <manojgupta@google.com> Tested-by: Manoj Gupta <manojgupta@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-11-01Update formatting of files before committing changes for autotest. No ↵Manoj Gupta
functional change. BUG: chromium:647429 TEST: crosperf unit tests Change-Id: I827f9ec4d00a0d8c1ae8a9930d7832289e694dc8 Reviewed-on: https://chrome-internal-review.googlesource.com/301457 Commit-Ready: Manoj Gupta <manojgupta@google.com> Tested-by: Manoj Gupta <manojgupta@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-09-13crosperf+cros_utils: Swap to #!/usr/bin/env.George Burgess IV
`cros lint` was recently updated to complain about using #!/usr/bin/python{,2,3} directly. Instead, it prefers /usr/bin/env python{2,3} now. BUG=None TEST=./run_tests.sh passes; linter seems much happier with crosperf. Change-Id: I9b88a7af1f8e03b7c870781c3d8dc1b872a3dfc3 Reviewed-on: https://chrome-internal-review.googlesource.com/286341 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-07-29toolchain-utils: fix results_cache_unittest.pyRahul Chaudhry
The test was failing with "RuntimeError: Could not run find command!". It broke after https://chrome-internal-review.googlesource.com/#/c/270698 which added GetResultsFile(). This CL adds the Fake version for testing. BUG=None TEST='./results_cache_unittest.py passes' Change-Id: I2bb92a394c8756bfb30941ce12e87396b22efe02 Reviewed-on: https://chrome-internal-review.googlesource.com/272168 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-04-14[crosperf] Fix more lint errors.Caroline Tice
BUG=chromium:567921 Test=Tested in my directory. Change-Id: Ibe128dd030887e0372380fb8f95a5e37ee63feeb Reviewed-on: https://chrome-internal-review.googlesource.com/255303 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-01-26crosperf: Make things work without access to file locks dirDavid Sharp
Before this change, running with --dry_run would result in an error accessing the lock file directory if the user does not have access to the directory. Also disable using file locks in machine_manager_unittest TEST=run crosperf with --dry_run; unit tests BUG=none Change-Id: I8283bc38fd388dfa90fb772e76370a674e86b86f Reviewed-on: https://chrome-internal-review.googlesource.com/245899 Commit-Ready: David Sharp <dhsharp@google.com> Tested-by: David Sharp <dhsharp@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-30crosperf: fix lint warnings.Yunlian Jiang
BUG=chromium:570454 TEST=unittest passes, lint warning is gone. Change-Id: I37f71a1195454657f205845d346b5ad76be878ec Reviewed-on: https://chrome-internal-review.googlesource.com/243141 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-29crosperf: fix lint warnings.Yunlian Jiang
BUG=chromium:570454 TEST=the lint warning is gone, unittest still passes. Change-Id: Icdb9c0ba6ecdd7a04bec8eafc96f84813e59b583 Reviewed-on: https://chrome-internal-review.googlesource.com/243139 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-29crosperf: fix lint warnings.Yunlian Jiang
BUG=chromium:570454 TEST=fixed some lint warnings. Change-Id: I645331f5d997cf7b470792239762fc0b78b21261 Reviewed-on: https://chrome-internal-review.googlesource.com/243239 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-22crosperf: fix lint warnings.Yunlian Jiang
BUG=none TEST=unittest passes. Change-Id: I1fb0c8c3fbf31fb4a26b7e02c34207f16541e0b5 Reviewed-on: https://chrome-internal-review.googlesource.com/243086 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-17Fix unittests that recent changes broke.Caroline Tice
BUG=chromium:570522 TEST=Tested in my directory. Change-Id: I197f5f5be3399babc838f4345173625e4d5bd91c Reviewed-on: https://chrome-internal-review.googlesource.com/242565 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-15crosperf: RunCommand should return one type of object.Luis Lozano
Cleaned up the interfaces for the RunCommand routines. These were returning different types (int or tuple) depending on the value of the return_ouput parameter. Returning different unrelated types from a routine is bad practice. Linter complains about this with several warnings like this: "Attempting to unpack a non-sequence defined at line XY of utils.command_executer" BUG=chromium:566256 TEST=ran crosperf with a example experiment file Ran run_tests. Change-Id: Ibb83ab9322c87558077fc4937ef5c0686bbe5417 Reviewed-on: https://chrome-internal-review.googlesource.com/241459 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Han Shen <shenhan@google.com>
2015-12-14Crosperf: Finish fixing unittest problems.Caroline Tice
This patch finishes fixing the unittest problems. They now all work. BUG=chromium:538397 TEST=./run_tests.sh now completes with no errors. Change-Id: Ie501437d199bfafc8a84cd2a95862550992c2cb8 Reviewed-on: https://chrome-internal-review.googlesource.com/241925 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-09crosperf: use cros_utils instead of utils.Yunlian Jiang
We got some lint warnings about some module is not in utils. So we change the name. This CL creates a symbolic link cros_utils from utils. Once we are sure that we clean all the code, we can rename the utils to cros_utils directly. BUG=chromium:567921 chromium:568195 TEST=the lint warning is gone. Change-Id: I776e45c7bdd75d1f065ea5797f3bcff457203ca2 Reviewed-on: https://chrome-internal-review.googlesource.com/241457 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-07-16Add 'run_local' to run the test harness locally.Ting-Yuan Huang
The option 'run_local: True|False' will be passed into autotest. For example, this is how test_that will be invoked: test_that --arges="run_local=True ..." ... BUG=None TEST=Tested with the folllowing combinations. benchmark: sunspider, smoothness.top_25_smooth perf_args: record -e cycles,instructions / (none) Change-Id: I7315027a7c9433d17a9f3fe54d7e8c3f480ea4f4 Reviewed-on: https://chrome-internal-review.googlesource.com/217370 Reviewed-by: Caroline Tice <cmtice@google.com> Tested-by: Ting-Yuan Huang <laszio@google.com> Commit-Queue: Ting-Yuan Huang <laszio@google.com>
2015-05-22Remove run_remote_tests references.cmtice
run_remote_tests.sh is about to be removed from ChromeOS. This CL removes all references to run_remote_tests from toolchain-utils. It also updates unittests to make sure they work with recent changes. BUG=crbug:498778 TEST=Verifed results_cache_unittest (and all the other unittests) works with changes. Change-Id: Ic9413bf77a1679e1267b81b182fa7f64da2629b5 Reviewed-on: https://chrome-internal-review.googlesource.com/216884 Reviewed-by: David Sharp <dhsharp@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-10-22Fix issue where crosperf can't find local cros_scratch.cmtice
Crosperf was assuming the user always has an NFS home directory, and was looking for the cros_scratch directory there. This makes crosperf use the user's home directory, whether it's NFS or local. Also change --share_users, which also assumed NFS home directories, to --share_cache, which requires specification of the cache path(s) to use. BUG=None TEST=Tested this with my new local homedir. Change-Id: I28d15467dfbd25b8b237cefa2979d3fd9464a7a7 Reviewed-on: https://chrome-internal-review.googlesource.com/179978 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-07-09Update and expand regression tests for results_cache.pycmtice
BUG=None TEST=Ran regression tests successfully. Ran 'normal' crosperf tests successfully as well. Change-Id: Ib3387cb63f1050cc0d561cc676d91be2334e0c79 Reviewed-on: https://chrome-internal-review.googlesource.com/166133 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-06-02Add --cache_only global setting to crosperf.cmtice
The CL adds a new setting, --cache_only, which causes crosperf to ONLY report test results that it finds in the cache (it will NOT rerun any tests, even if no cache results are found). This is partly for implementing a weekly summary report, where we want to compare the nightly results from the past week, but not actually rerun anything. BUG=None TEST=I tested this option while generating weekly test reports, and it worked as it is supposed to. Change-Id: Icb1f5343f76ce999bebd0810bb86ab64a7577313 Reviewed-on: https://chrome-internal-review.googlesource.com/164890 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-21Clean up Crosperf options and help.cmtice
Crosperf's help was completely out of date and incorrect. Fixed that. Also, I cleaned up the options, removed some that were no longer appropriate or not ever used, and made some options global-only. Also removed all references to pyauto. We no longer use or run pyauto tests. Option changes: Completely removed outlier_range (it was not actually doing anything), key_results_only (applied only to pyauto tests), use_test_that (not needed now that we're not using pyauto) and md5sum (never used). Made the following options global-only: rm_chroot_tmp, perf_args, logging_level, show_all_results, board. Removed 'build' from global settings and made it an image-only setting (it made no sense as a global setting). I also adjusted the unittests to work with these changes. BUG=None TEST=I tested all the options in all their various positions; I tested with and wtihout caching; and I tested local and official images. I also ran the unittests. Everything works. Change-Id: I7653663f65c0a3363a1904c0220f1066e72c7f9d Reviewed-on: https://chrome-internal-review.googlesource.com/160758 Reviewed-by: Bhaskar Janakiraman <bjanakiraman@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-11Update unittests to all pass.cmtice
Fix the parameters to the various unittests so they match recent changes and the unittests all pass again. BUG=None TEST=I ran all the unittests with the changes. Change-Id: I083b5127a2ade8f1dbaf2bb173d82183871cb7c7 Reviewed-on: https://chrome-internal-review.googlesource.com/159915 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-12-10Use default results for Telemetry tests; add option to use all results.cmtice
When running telemetry_Crosperf, use the json file of report defaults to determine which results to actually include in the report. This CL also adds a flag to Crospref, --show_all_results=True, that allows the user to override the defaults and generate a report showing all the values. BUG=None TEST=Tested this both with and without the --show_all_results flag. It works as expected for me. Change-Id: I91364572ebdf3d54bb8b408fbd5a2580923f6829 Reviewed-on: https://chrome-internal-review.googlesource.com/149457 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-10-01crosperf: fix unittest for crosperf.Yunlian Jiang
This fixes several issues for unittesting. Now unittest seems to be working. BUG=None TEST=crosperf/run_tests.sh passes. Change-Id: I1dcef7ba55d45fbd00e453f749a8b10dea0332e6 Reviewed-on: https://chrome-internal-review.googlesource.com/145635 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com>
2013-05-22crosperf: add results_cache_unittestYunlian Jiang
BUG=None TEST=this unittest passes. Change-Id: I65fee19efd30b1602a6ad49b84abe7366a6e605c Reviewed-on: https://gerrit-int.chromium.org/38429 Reviewed-by: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Yunlian Jiang <yunlian@google.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-11 11:09:50 +0000