aboutsummaryrefslogtreecommitdiff
path: root/crosperf
AgeCommit message (Collapse)Author
2020-02-07crosperf: introduce new test_that and skylab run for suite_runnerZhizhou Yang
With client tests wrapped up by crosperf_Wrapper server tests, we need a categorizing for all tests lunched by suite runner. BUG=chromium:984790 TEST=Passed unittest and simple experiment. Change-Id: Ib7dc848cf5c18c3d79db059163ed1217fd7d318f Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1977040 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2020-01-30crosperf: migration to python 3Zhizhou Yang
This patch migrates crosperf and its utils to python 3. TEST=Passed presubmit check; tested with simple experiment locally. BUG=chromium:1011676 Change-Id: Ib2a9f9c7cf6a1bb1d0b42a1dd3d9e3cbb4d70a36 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2003796 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2020-01-24crosperf: remove AFE server interactionsZhizhou Yang
This patch removes afe locking mode for crosperf, since: 1) All lab machines has been migrated to skylab. 2) Interaction with afe depends on python 2 code in autotest. BUG=chromium:1044764 TEST=Passed unittest and simple experiment run. Change-Id: Ie0a250548d739f32bbc08619f637380104df6c28 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2016008 Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: George Burgess <gbiv@chromium.org>
2020-01-16crosperf: fix pylint for crosperfZhizhou Yang
This is the first step of python 3 migration for crosperf TEST=Passed presubmit for all .py files in crosperf BUG=chromium:1011676 Change-Id: I615d0323c81ba688c8ce2aa0cac307035d0b882b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2001696 Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-12-14crosperf: fix skylab telemetry_Crosperf tests launching commandZhizhou Yang
Current command generated from suite_runner for telemetry_Crosperf run is not correct, and this patch will fix it. Also added an unittest to verify it in the future. It is unfortunate that we currently still cannot run test with args successfully in skylab, but according to the args passed to skylab, this change should work. TEST=Checked command line to launch skylab create-test. BUG=None Change-Id: I05f0f4c60e04d19e2a2637192023829d4dd9f48b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1963355 Reviewed-by: George Burgess <gbiv@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-12-12Crosperf: Do not image DUT in Skylab modeZhizhou Yang
We should not image DUT with certain image when the test is going to be launched via 'skylab create-test'. This is because the DUT may be occupied by other tasks before Crosperf acutally get it assigned, and the first step after the test starts is to flash image. This patch is a temproray solution to ensure no bad behavior happens on the DUT before testing. TEST=Tested wth skylab run, w/o caching. BUG=chromium:984790 Change-Id: Ie2f439d5777813905bdca5f4a2d1e401df2dc83e Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1952172 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-12-11crosperf: migrate device setup functions for telemetry_CrosperfZhizhou Yang
This is a patch working together with crrev.com/c/1949606 to migrate device setup functions from suite_runner to autotest telmetry_Crosperf. In this patch, we modified the logic of different suite types along with skylab mode for suite_runner. In result_cache, we handle the wait time log file to accumulate cool down time per machine. We also move intel_pstate kernel updating code into SetupDevice() since it doesn't increase overhead by checking every iteration. TEST=Passed all unittests; tested with different benchmark modes (Note that currently skylab server tests cannot be tested because of regression: crbug.com/984103). BUG=chromium:1020655 Cq-Depend: chromium:1949606 Change-Id: If8e5099d096d2568d4f54584a8fcfd3c0f99c8f8 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1949626 Reviewed-by: George Burgess <gbiv@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Commit-Queue: Denis Nikitin <denik@chromium.org>
2019-12-05crosperf: Filter-in exp_avg_surface_fps in rendering.desktopDenis Nikitin
Rendering.desktop benchmark which was added into crosperf does not return results in avg_surface_fps field. Until recently it worked fine. There is an alternative field in histograms.json "exp_avg_surface_fps" holding the data. Add exp_avg_surface_fps in default-telemetry-results.json to filter-in the benchmark field in crosperf results. BUG=chromium:1030475 TEST=crosperf with rendering.desktop returns data on chell and lulu Change-Id: Ia1d25f7f1894ca559f1e14e032fcf274dc4df49d Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1951336 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-12-04crosperf: move enable_aslr and turbostat option into dut_configZhizhou Yang
This patch moves enable_aslr and turbostat into dut_config dictionary, since they are all device setup steps, and it make code cleaner and easier to migrate to telemetry_Crosperf. TEST=Passed unittests. BUG=chromium:1020655 Change-Id: I29a649e76591e206efc19bb2b8a8df9aead16575 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1947903 Reviewed-by: Denis Nikitin <denik@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-12-03crosperf: Make StopUI/StartUI more robustDenis Nikitin
There was a bug when crosperf failed due to StopUI failure. This happened when in a preceding run crosperf exits with an exception in DeviceSetup (for example Keyboard interrupt) and fails to call StartUI. Current change fixes this problem and includes a unittest case testing the exception case. BUG=None TEST=unittest and HW test on DUT with stopped ui pass Change-Id: Id6c69aebefe21b12ec7ee3a7c7f9dff92d143908 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1941036 Commit-Queue: Denis Nikitin <denik@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2019-11-25crosperf: fix the skylab launch tests modeZhizhou Yang
Crosperf provides a mode to use `skylab create-test` to launch tests when user specify 'skylab: true' in experiment file. Since Skylab has been updated in these months, the command line itself changes. This patch fixes it: 1) bb mode becomes deprecated, do not use it. 2) Task id becomes a recipe id, we need to parse it instead of swarming. 3) Real result location can be found in the json output of `wait-task` from key "child-result". TEST=Passed unittest, tested with simple experiment file. BUG=chromium:984790 Change-Id: Iee114b1e6c623bb64a0ca784df9c2ba6d95e3d07 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1916542 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-11-13crosperf: fix unittest error without prodaccessZhizhou Yang
After locking mechanism changed, crosperf checks if locks_dir is set or not and by default uses X20 location. This requires prodaccess thus unittests will fail without prodaccess if locks_dir is not explicitly set. This patch set all experiment files in unittests to use /tmp for locks_dir. Also fixed some lint error for deprecated functions. TEST=Passed all unittest. BUG=None Change-Id: Iff4623aad9632842010e5efb85a1e75a38bcf73e Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1912767 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-11-12crosperf: Update xbuddy location.Manoj Gupta
xbuddy location has changed to chromite. So adjust crosperf accordingly. BUG=b:144248787 TEST=translate_xbuddy works. Change-Id: Id7b7542d485f906fdcad7918f916b42286ce961a Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1911381 Tested-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2019-11-12crosperf: migrate all device setup code to a separate utils fileZhizhou Yang
This patch extracts all the device setup code which interacts with DUT to a single utils file, and be put into a wrapper class. This will help migrating all related code to telemetry_Crosperf for skylab runs. BUG=chromium:1020655 TEST=Passed all unittests; tested with simple experiment on kevin. Change-Id: I2edcd7bb2d8cd0255d3ae6d380a5983c24427d98 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1895500 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Denis Nikitin <denik@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com>
2019-11-05crosperf: Improve cooldown functionalityDenis Nikitin
"Restart UI" procedure is changed. DUT configuration including cooldown in SuiteRunner is now performed while UI is off. Start UI after DUT configuration is done. The change showed decrease of cooldown waiting time by a factor of 10. Added Total experiment duration time in the report along with total cooldown wait time. Cooldown wait time is now printed per machine (with parallel execution on multiple machines single cooldown time could be misleading). To collect data per machine cooldown_time attribute is moved from SuiteRunner to CrosMachine class. BUG=chromium:966514 TEST=unittests and HW tests on kevin64 passed. Change-Id: Ib3e145cd986ca7242d97b0aeeb7795ba066753fc Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1897236 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-11-04crosperf: fix a download image bugZhizhou Yang
This is a fix to a bug introduced by https://chromium-review.googlesource.com/1865958. GetXbuddyPath() function is also called in experiment_file and it's for re-writing the fields to output. So at that time, we don't need to really check whether downloading debug again. BUG=chromium:1010294 TEST=tested with kevin locally. Change-Id: Ic7daf6d1242f012bdef803faca355df0365f2cdf Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1895979 Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org>
2019-10-27crosperf: Add option to not download debug symbolsZhizhou Yang
This patch introduces a new option --download_debug to crosperf. We will only download debug symbols when this option is set to True. This will save space when perf test users doesn't need it. TEST=passed unittests. BUG=chromium:1010294 Change-Id: Ic80958377f1c25cb04e3b7b56b40fafcada6d2d8 Reviewed-on: https://chromium-review.googlesource.com/1865958 Tested-by: Zhizhou Yang <zhizhouy@google.com> Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org>
2019-10-16crosperf: Update top stats and cooldown reportDenis Nikitin
Redirect top statistics from benchmark runs into a separate file topstats.log under results_dir directory. Fix "highest 5" usages to show highest usages of a command (instead of a process) per snapshot. Improve mechanism of calculation chrome high CPU load when benchmark is running. Add Cooldown wait time into email report. Fix minor cros lint warnings to unblock repo upload. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I3999efd554cb5a3b27a2ce3fddb2f20714b434fd Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1856818 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-10-15toolchain-utils: change naming related to AFE lockZhizhou Yang
After introducing the new locking mechanism, the behavior of AFELockManager totally changed. This file changed the naming of all related code in toochain-utils. This patch also changed the behavior of locks_dir option, deprecated the use_file_lock option explicitly. TEST=Tested with different DUT types. BUG=chromium:1006434 Change-Id: Ib15efce54ec4d4c5c2a18fecca3f350248462035 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1863530 Reviewed-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-10-09crosperf: gsutil location changedZhizhou Yang
Most recently, the checkout location of depot_tools in chromeos has been changed, which will break Crosperf at image downloading. This patch fixes it. BUG=chromium:1012904 TEST=Passed unittests. Change-Id: I60b8de6aa3a34faa8be06aedc8ff169da993d5a5 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1848858 Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-10-09crosperf: Change video crosbolt benchmark to Tast versionKeiichi Watanabe
Since video autotests are already ported to Tast and stopped running, replace crosbolt benchmark with Tast version. BUG=chromium:953553 TEST=None Change-Id: I09a44027ab2dc338679d90f4aa8ad362f160852e Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1801410 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2019-09-28crosperf: Print top5 commands into logDenis Nikitin
Enable top in nightly tests and print top 5 commands with high CPU usage into log. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I0efd7f0875829c4423e998ec2fbec7436f7ae148 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1829309 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-27toolchain-utils: remove AFE local server codeZhizhou Yang
We used to have local server for AFE locking mechanism to lock non-lab machines. But this local server was not supported since long time ago. This patch removes all local server related code from AFE locking. Note that there will be another CLs to deal with issues in http://crbug.com/1006434#c1 TEST=Passed crosperf local tests with different type of machines. BUG=chromium:1006434 Change-Id: I47dbca73edd3e493496fd13ba0b3de705dc11513 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1822029 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-09-26crosperf: Add cpu temp/freq in cwp reportDenis Nikitin
Result organizer with CWP has a different control flow and didn't include cpu frequency and temperature data. Current change adds missing data into the Full table of the report file. BUG=None TEST=tested on veyron with cwp experiment file. Change-Id: I67447ab016cee8265c1937b236cc5c38330576e9 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1827918 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2019-09-26crosperf: mark a run as failed if samples collected is 0Zhizhou Yang
There is situation that in cwp mode, test runs correctly but not samples are collected. For this case, we want to mark the run as failure so that we could have correct speedup calculation and user can notice it. BUG=chromium:927554 TEST=Passed unit tests Change-Id: I4d4f9bd25159e8d3d0b14a84f7f566593e1438ac Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1594288 Tested-by: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
2019-09-24crosperf: Fix cros lint warningsDenis Nikitin
Fixed python lint warnings in crosperf scripts. BUG=chromium:993035 TEST=`repo upload` works w/o --no-verify Change-Id: I3dd7b0ea956458e57e8bd77c19a5b8cb13e1d350 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1821449 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-24crosperf: add per tag story summary when parsing histogramsJian Cai
Add support to collecting results based on story tags in histograms. This allows us to remove the workaround for loading.desktop and deprecate chartjson. BUG=chromium:1000497 TEST=local tests. Change-Id: I8249d00b3d2d5d761c8a3bc24ef2fd95a3e1115b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1783837 Tested-by: Jian Cai <jiancai@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Tiancong Wang <tcwang@google.com>
2019-09-24crosperf: introducing new locking mechanismZhizhou Yang
In this patch, we modified the behavior of crosperf locking system. We no longer need to specify whether to use file locking or afe locking, instead locking methods are all automatically detected. We also introduced a new leasing mechanism for skylab duts. Now, if a dut: - in afe: use afe lock on it - in skylab: lease via skylab command - local: use file lock on it When crosperf finishes or dies, we will also try to unlock those locked by us. There will be another CL to change the naming of afe_lcok to general locks and remove unused afe local server. BUG=chromium:984790 TEST=Tested crosperf with DUTs in Skylab quota pool, afe and locally. Change-Id: I0a9bf4a16f54cfa0e8af077765fbe5a0a39e7c88 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1793911 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
2019-09-24crosperf: Add top_interval argument to experimentDenis Nikitin
Added the following optional argument in the global settings: "top_interval" - Run top command in the background of a benchmark with interval of sampling specified in seconds (float type). With zero value don't run top (default value). Top output is stored in <resultsdir>/top.log. BUG=chromium:966514 TEST=unittests and HW tests on eve, kevin64, scarlet passed. Change-Id: I87b187c85912140fc0c16e224580b60061c8bd94 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1817121 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-17crosperf: Fix thermal failure on veyronDenis Nikitin
Added the flag to ignore command error when requesting thermal data from DUT. It appeared that some devices (like veyron-minnie) have sensors with not readable temperature. BUG=None TEST=added unittest case and ran HW test on local veyron-minnie. Change-Id: I1edb12fce7a784a31f6fb93267c942d308fb2891 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1808021 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-16crosperf: Setup CPU frequencyDenis Nikitin
Added "cpu_freq_pct" in global settings specifying percent of the max frequency which will be set on DUT during benchmark run. The actual value will be chosen from available frequencies less than or equal to the calculated number. Feature requires scaling_available_frequencies. On the platforms not supporting scaling_available_frequencies "cpu_freq_pct" will be ignored. BUG=chromium:966514 TEST=Unittest and local HW tests on veyron, scarlet and kevin64 passed. Change-Id: I175e2836ecf02fcf6b93d196926c7801d7f016c0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1803780 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-09-13crosperf: Fix unittest execution timeDenis Nikitin
Added mock patches for time.time() and time.sleep() in experiment_runner_unittest.py. Unittest execution time dropped from 12s to <1s. BUG=chromium:989571 TEST=cd <path-to-crosperf>; ./run_tests.sh Change-Id: I60af5305cd022b9a5854c444cfa9c50cbe0a22c0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1801423 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-09-13crosperf: Wait for CPU cooldown and setup governor.Denis Nikitin
Based on dut_config['cooldown_temp'] and dut_config['cooldown_time'] wait for CPU to cooldown to a temperature 'cooldown_temp' or until timeout 'cooldown_time' expires. While waiting change CPU governor to a power saving mode. With 'cooldown_time' = 0 don't wait. Setup CPU governor based on dut_config['governor'] for a benchmark run. Additionally function helper DutWrapper is added in SuiteRunner which simplifies remote DUT calls. BUG=chromium:966514 TEST=Unitest and local HW tests passed. Change-Id: I2b946b570e7a8d8c4e2321f8bb8de3be2c3162d0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1790131 Tested-by: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-10crosperf: Setup CPU configuration based on 'cpu_usage'.Denis Nikitin
Based on dut_config['cpu_usage'] setup DUT to use big or little cores only on ARM platform. Ignore 'cpu_usage' if setup is invalid for targeted platform. For example 'big_only" on Intel or ARM w/o big/little support. BUG=chromium:966514 TEST=Unitest and local HW tests passed. Cq-Depend: chromium:1778515 Change-Id: Ida457a1ceaf20c4dcbb8d40334a2423836999574 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1783849 Reviewed-by: Caroline Tice <cmtice@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-09-10crosperf: Add dut config arguments to experimentDenis Nikitin
In global settings added optional arguments: "cooldown_time" - wait time prior running a benchmark (default: 0), "cooldown_temp" - temperature threshold for waiting (default: 40), "governor" - CPU governor (default: performance), "cpu_usage" - Restrict CPU usage to specific configurations (default: all). "turbostat" argument is moved from benchmark to global settings. Current CL does not apply configurations. Instead it just propagates parameters to SuiteRunner class through "dut_config" dictionary. BUG=chromium:966514 TEST=Unitest and local HW tests passed. Change-Id: I1b2a65883e5176fdde49c9858ebe62a097df89cb Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1778515 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-09-03crosperf: update skylab command lineZhizhou Yang
Recently there are some big changes to options of skylab command, this patch updates the command to the newest version. BUG=chromium:984790 TEST=Tested with different crosperf tests. Change-Id: I6ed81c00372668914f79b66c1e18d980c073593d Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1775533 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-30crosperf: Add support for using aquarium in Telemetry.Tiancong Wang
We want to test the story aquarium in rendering.desktop and see the results comparison with graphics_WebGLAquarium in autotest. This patch adds support to filter out the most important metric to display in the report for the two aquarium stories (with different number of fish). Also fix the logic of reading histogram results when there's None value in the list. Also add the test to toolchain nightly tests. BUG=None TEST=crosperf --board=nami with the two benchmarks Change-Id: Ia1aa6bf796bee78bf6199332a07f4b96aa084845 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1779248 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-26crosperf: Add jetstream2 to telemetry perf tests.Tiancong Wang
As jetstream2 is added to telemetry benchmarks, we want to add it to the list of telemetry toolchain perf tests, and add supports in crosperf. Also fixed some formatting problems of the file. BUG=None TEST=jetstream2 run locally, will be tested more on nightly tests. Change-Id: I89f748ca8d4a194406688733e91cca47f8a3f5fe Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1769482 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-19crosperf: Add ARM CPU stats in crosperf reportDenis Nikitin
Check for cpustats.log file in results_dir and extract data with CPU frequncy and temperature. Calculate avg/min/max values from the measurements and add to result keyvals. The parser attempts to read turbostat.log first (which comes from Intel devices along with cpustats.log) and only if it doesn't exist or empty switches to cpustats.log (usually ARM devices). BUG=chromium:966514 TEST=tested locally on veyron, scarlet, kevin64 (arm), eve (intel) Change-Id: I12991b39a0f1acb6197ca663a6876b0f013d2026 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1753164 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-08-09crosperf: Print traceback instead of exception type.Tiancong Wang
I found that the error message of crosperf is sometimes frustrating that when you trigger an exception somewhere in the crosperf tool, it will always return the exception type. Sometimes it's really hard to find out where the exception/error happens in your code. I think it might make sense to handle all exceptions with a try...except block, to make the error message always appear at last, so raising the original exception prints whole traceback, instead of dropping it to general SystemExit. BUG=None TEST=Tested locally, and added unittest. Change-Id: Id772af444bf3788011c74504c71939d2514f1752 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1743146 Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-08-07toolchain-utils: Fix argument-passing error in experiment_factory.pyCaroline Tice
Fix all calls to Benchmark inside experiment_factory.py to pass all args, to prevent an arg with a default value accidentally being passed a value for a different arg. BUG=chromium:991396 TEST=Tested locally with Crosperf and it worked properly. Change-Id: Ia8096e2989e179723e122090ea40a11747070b22 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1740646 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Tested-by: Caroline Tice <cmtice@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
2019-08-02crosperf: Add "intel_pstate" and "turbostat" argsDenis Nikitin
In global setting added "intel_pstate" option. Type: string. Values: active, passive, no_hwp, ''(equivalent to active). Default ''. In benchmark setting added "turbostat" option. Type: boolean. Default True. BUG=chromium:966514 TEST=tested on samus, eve, rammus Change-Id: I1533c6d6586aab5acf3665bce68920a0ab3092bc Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1726635 Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: Sean Abraham <seanabraham@chromium.org> Reviewed-by: Sean Abraham <seanabraham@chromium.org> Tested-by: Sean Abraham <seanabraham@chromium.org>
2019-08-02crosperf: Setup intel_pstateDenis Nikitin
Setup intel_pstate in kernel command line to a value from global settings "intel_pstate" argument (default ''). Added crosperf "--no_hwp" argument for a shortcut change. Kernel command line will be updated only if hwp is supported by CPU. Intel_pstate HWP feature can be disabled via kernel command line argument intel_pstate=no_hwp. See https://www.kernel.org/doc/html/v4.12/admin-guide/pm/ intel_pstate.html#kernel-command-line-options-for-intel-pstate BUG=chromium:966514 TEST=tested on eve, rammus (supported), kefka, samus (not supported), scarlet (arm). Change-Id: Id6b7cf7e230e17788c1024da2aacbc9a16b7ac8e Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1711517 Commit-Queue: Denis Nikitin <denik@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org>
2019-08-01crosperf: check skylab tool exists or not for skylab modeZhizhou Yang
This patch checks whether skylab tool is installed when skylab mode is specified. After installing it, it requires user to login the tool manually for the first time since there is no way to do it automatically. BUG=chromium:984790 TEST=Tested locally and passed unittest. Change-Id: Iff03ed0552127837264d8437e2b73346f1abdbd4 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1727264 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-07-28crosperf: Add cpu statistics into results reportDenis Nikitin
Check for turbostat.log file in results directory and if file exists parse the context and extract measurements of CPU Frequency and temperature for ALL cores (not each). Keys added to keyvals: 'cpufreq_min', 'cpufreq_max', 'cpufreq_avg', 'cputemp'. BUG=chromium:966514 TEST=tested all crosbolt perf benchmarks Change-Id: If90d26664c70ccb73750f17573b89933fdb048f4 Reviewed-on: https://chromium-review.googlesource.com/1709824 Tested-by: Denis Nikitin <denik@chromium.org> Commit-Ready: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-07-26crosperf: Fix pylint warningsDenis Nikitin
Pylint throws a bunch of warnings on results_cache.py and results_organizer.py (other source files not checked). The change fixes most warnings EXCEPT docstring which would require more elaboration. BUG=none TEST=presubmit passess Change-Id: I4e48d94c9b430d8b224e70e9267abe3795bff7cc Reviewed-on: https://chromium-review.googlesource.com/1719770 Tested-by: Denis Nikitin <denik@chromium.org> Commit-Ready: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-07-25toolchain-utils: add block list for histograms outputJian Cai
Histograms of some tests such as loading.desktop are very different from their chart-json counterpart. It seems it is almost impossible to parse these tests without case-by-case parsing. Parse chart-json output of these tests instead. BUG=chromium:984713 TEST=local tests Change-Id: I225567a1be9d401b5dd0c1f682184e581ac4acce Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1714125 Tested-by: Jian Cai <jiancai@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-07-25Toolchain-utils: update crosbolt listJian Cai
Move loading.desktop and rendering.desktop from crosbolt_perf_tests to telemetry_crosbolt_perf_tests. Otherwise test_that does not pass correct suite type. BUG=chromium:970061 TEST=local tests Change-Id: I485335fd285d0e132e0d7a2fe309a6303931de56 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1717127 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Jian Cai <jiancai@google.com>
2019-07-25Toolchain-utils: replace xrange with range on results_cache.pyJian Cai
xrange builtint is replaced by range on Python3. Use range instead of xrange on results_cache.py. BUG=chromium:986617 TEST=local tests Change-Id: I729b12a92b75dbd355fad3012205765a648be538 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1717449 Tested-by: Jian Cai <jiancai@google.com> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-07-25toolchain-utils: remove all xrangesGeorge Burgess IV
This removes all mention of xrange from toolchain-utils (modulo ones being changed in other CLs that are in flight). It's now an apparent lint error to use xrange, and it hinders our move to python3. As commented on If90d26664c70ccb73750f17573b89933fdb048f4, xrange -> range in python2 is really only a space concern (or speed in pathological cases), so migrations of this nature are generally super straightforward. I glanced at each of these callsites, and none of them appear to be pathological, so my hope is that this should all be Just Fine :) (Also fun to note that this includes a .diff file that has python code embedded in it.) BUG=None TEST=Presubmit tests Change-Id: Ic9f3ac3a5044d7a07da8a249bc505278d98203de Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1717130 Commit-Queue: George Burgess <gbiv@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: George Burgess <gbiv@chromium.org>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-11 05:49:34 +0000