aboutsummaryrefslogtreecommitdiff
path: root/crosperf
AgeCommit message (Collapse)Author
2019-09-24crosperf: Fix cros lint warningsDenis Nikitin
Fixed python lint warnings in crosperf scripts. BUG=chromium:993035 TEST=`repo upload` works w/o --no-verify Change-Id: I3dd7b0ea956458e57e8bd77c19a5b8cb13e1d350 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1821449 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-24crosperf: add per tag story summary when parsing histogramsJian Cai
Add support to collecting results based on story tags in histograms. This allows us to remove the workaround for loading.desktop and deprecate chartjson. BUG=chromium:1000497 TEST=local tests. Change-Id: I8249d00b3d2d5d761c8a3bc24ef2fd95a3e1115b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1783837 Tested-by: Jian Cai <jiancai@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Tiancong Wang <tcwang@google.com>
2019-09-24crosperf: introducing new locking mechanismZhizhou Yang
In this patch, we modified the behavior of crosperf locking system. We no longer need to specify whether to use file locking or afe locking, instead locking methods are all automatically detected. We also introduced a new leasing mechanism for skylab duts. Now, if a dut: - in afe: use afe lock on it - in skylab: lease via skylab command - local: use file lock on it When crosperf finishes or dies, we will also try to unlock those locked by us. There will be another CL to change the naming of afe_lcok to general locks and remove unused afe local server. BUG=chromium:984790 TEST=Tested crosperf with DUTs in Skylab quota pool, afe and locally. Change-Id: I0a9bf4a16f54cfa0e8af077765fbe5a0a39e7c88 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1793911 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
2019-09-24crosperf: Add top_interval argument to experimentDenis Nikitin
Added the following optional argument in the global settings: "top_interval" - Run top command in the background of a benchmark with interval of sampling specified in seconds (float type). With zero value don't run top (default value). Top output is stored in <resultsdir>/top.log. BUG=chromium:966514 TEST=unittests and HW tests on eve, kevin64, scarlet passed. Change-Id: I87b187c85912140fc0c16e224580b60061c8bd94 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1817121 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-17crosperf: Fix thermal failure on veyronDenis Nikitin
Added the flag to ignore command error when requesting thermal data from DUT. It appeared that some devices (like veyron-minnie) have sensors with not readable temperature. BUG=None TEST=added unittest case and ran HW test on local veyron-minnie. Change-Id: I1edb12fce7a784a31f6fb93267c942d308fb2891 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1808021 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-16crosperf: Setup CPU frequencyDenis Nikitin
Added "cpu_freq_pct" in global settings specifying percent of the max frequency which will be set on DUT during benchmark run. The actual value will be chosen from available frequencies less than or equal to the calculated number. Feature requires scaling_available_frequencies. On the platforms not supporting scaling_available_frequencies "cpu_freq_pct" will be ignored. BUG=chromium:966514 TEST=Unittest and local HW tests on veyron, scarlet and kevin64 passed. Change-Id: I175e2836ecf02fcf6b93d196926c7801d7f016c0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1803780 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-09-13crosperf: Fix unittest execution timeDenis Nikitin
Added mock patches for time.time() and time.sleep() in experiment_runner_unittest.py. Unittest execution time dropped from 12s to <1s. BUG=chromium:989571 TEST=cd <path-to-crosperf>; ./run_tests.sh Change-Id: I60af5305cd022b9a5854c444cfa9c50cbe0a22c0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1801423 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-09-13crosperf: Wait for CPU cooldown and setup governor.Denis Nikitin
Based on dut_config['cooldown_temp'] and dut_config['cooldown_time'] wait for CPU to cooldown to a temperature 'cooldown_temp' or until timeout 'cooldown_time' expires. While waiting change CPU governor to a power saving mode. With 'cooldown_time' = 0 don't wait. Setup CPU governor based on dut_config['governor'] for a benchmark run. Additionally function helper DutWrapper is added in SuiteRunner which simplifies remote DUT calls. BUG=chromium:966514 TEST=Unitest and local HW tests passed. Change-Id: I2b946b570e7a8d8c4e2321f8bb8de3be2c3162d0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1790131 Tested-by: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-10crosperf: Setup CPU configuration based on 'cpu_usage'.Denis Nikitin
Based on dut_config['cpu_usage'] setup DUT to use big or little cores only on ARM platform. Ignore 'cpu_usage' if setup is invalid for targeted platform. For example 'big_only" on Intel or ARM w/o big/little support. BUG=chromium:966514 TEST=Unitest and local HW tests passed. Cq-Depend: chromium:1778515 Change-Id: Ida457a1ceaf20c4dcbb8d40334a2423836999574 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1783849 Reviewed-by: Caroline Tice <cmtice@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-09-10crosperf: Add dut config arguments to experimentDenis Nikitin
In global settings added optional arguments: "cooldown_time" - wait time prior running a benchmark (default: 0), "cooldown_temp" - temperature threshold for waiting (default: 40), "governor" - CPU governor (default: performance), "cpu_usage" - Restrict CPU usage to specific configurations (default: all). "turbostat" argument is moved from benchmark to global settings. Current CL does not apply configurations. Instead it just propagates parameters to SuiteRunner class through "dut_config" dictionary. BUG=chromium:966514 TEST=Unitest and local HW tests passed. Change-Id: I1b2a65883e5176fdde49c9858ebe62a097df89cb Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1778515 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-09-03crosperf: update skylab command lineZhizhou Yang
Recently there are some big changes to options of skylab command, this patch updates the command to the newest version. BUG=chromium:984790 TEST=Tested with different crosperf tests. Change-Id: I6ed81c00372668914f79b66c1e18d980c073593d Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1775533 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-30crosperf: Add support for using aquarium in Telemetry.Tiancong Wang
We want to test the story aquarium in rendering.desktop and see the results comparison with graphics_WebGLAquarium in autotest. This patch adds support to filter out the most important metric to display in the report for the two aquarium stories (with different number of fish). Also fix the logic of reading histogram results when there's None value in the list. Also add the test to toolchain nightly tests. BUG=None TEST=crosperf --board=nami with the two benchmarks Change-Id: Ia1aa6bf796bee78bf6199332a07f4b96aa084845 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1779248 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-26crosperf: Add jetstream2 to telemetry perf tests.Tiancong Wang
As jetstream2 is added to telemetry benchmarks, we want to add it to the list of telemetry toolchain perf tests, and add supports in crosperf. Also fixed some formatting problems of the file. BUG=None TEST=jetstream2 run locally, will be tested more on nightly tests. Change-Id: I89f748ca8d4a194406688733e91cca47f8a3f5fe Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1769482 Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-19crosperf: Add ARM CPU stats in crosperf reportDenis Nikitin
Check for cpustats.log file in results_dir and extract data with CPU frequncy and temperature. Calculate avg/min/max values from the measurements and add to result keyvals. The parser attempts to read turbostat.log first (which comes from Intel devices along with cpustats.log) and only if it doesn't exist or empty switches to cpustats.log (usually ARM devices). BUG=chromium:966514 TEST=tested locally on veyron, scarlet, kevin64 (arm), eve (intel) Change-Id: I12991b39a0f1acb6197ca663a6876b0f013d2026 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1753164 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-08-09crosperf: Print traceback instead of exception type.Tiancong Wang
I found that the error message of crosperf is sometimes frustrating that when you trigger an exception somewhere in the crosperf tool, it will always return the exception type. Sometimes it's really hard to find out where the exception/error happens in your code. I think it might make sense to handle all exceptions with a try...except block, to make the error message always appear at last, so raising the original exception prints whole traceback, instead of dropping it to general SystemExit. BUG=None TEST=Tested locally, and added unittest. Change-Id: Id772af444bf3788011c74504c71939d2514f1752 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1743146 Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-08-07toolchain-utils: Fix argument-passing error in experiment_factory.pyCaroline Tice
Fix all calls to Benchmark inside experiment_factory.py to pass all args, to prevent an arg with a default value accidentally being passed a value for a different arg. BUG=chromium:991396 TEST=Tested locally with Crosperf and it worked properly. Change-Id: Ia8096e2989e179723e122090ea40a11747070b22 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1740646 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Tested-by: Caroline Tice <cmtice@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
2019-08-02crosperf: Add "intel_pstate" and "turbostat" argsDenis Nikitin
In global setting added "intel_pstate" option. Type: string. Values: active, passive, no_hwp, ''(equivalent to active). Default ''. In benchmark setting added "turbostat" option. Type: boolean. Default True. BUG=chromium:966514 TEST=tested on samus, eve, rammus Change-Id: I1533c6d6586aab5acf3665bce68920a0ab3092bc Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1726635 Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: Sean Abraham <seanabraham@chromium.org> Reviewed-by: Sean Abraham <seanabraham@chromium.org> Tested-by: Sean Abraham <seanabraham@chromium.org>
2019-08-02crosperf: Setup intel_pstateDenis Nikitin
Setup intel_pstate in kernel command line to a value from global settings "intel_pstate" argument (default ''). Added crosperf "--no_hwp" argument for a shortcut change. Kernel command line will be updated only if hwp is supported by CPU. Intel_pstate HWP feature can be disabled via kernel command line argument intel_pstate=no_hwp. See https://www.kernel.org/doc/html/v4.12/admin-guide/pm/ intel_pstate.html#kernel-command-line-options-for-intel-pstate BUG=chromium:966514 TEST=tested on eve, rammus (supported), kefka, samus (not supported), scarlet (arm). Change-Id: Id6b7cf7e230e17788c1024da2aacbc9a16b7ac8e Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1711517 Commit-Queue: Denis Nikitin <denik@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org>
2019-08-01crosperf: check skylab tool exists or not for skylab modeZhizhou Yang
This patch checks whether skylab tool is installed when skylab mode is specified. After installing it, it requires user to login the tool manually for the first time since there is no way to do it automatically. BUG=chromium:984790 TEST=Tested locally and passed unittest. Change-Id: Iff03ed0552127837264d8437e2b73346f1abdbd4 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1727264 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-07-28crosperf: Add cpu statistics into results reportDenis Nikitin
Check for turbostat.log file in results directory and if file exists parse the context and extract measurements of CPU Frequency and temperature for ALL cores (not each). Keys added to keyvals: 'cpufreq_min', 'cpufreq_max', 'cpufreq_avg', 'cputemp'. BUG=chromium:966514 TEST=tested all crosbolt perf benchmarks Change-Id: If90d26664c70ccb73750f17573b89933fdb048f4 Reviewed-on: https://chromium-review.googlesource.com/1709824 Tested-by: Denis Nikitin <denik@chromium.org> Commit-Ready: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-07-26crosperf: Fix pylint warningsDenis Nikitin
Pylint throws a bunch of warnings on results_cache.py and results_organizer.py (other source files not checked). The change fixes most warnings EXCEPT docstring which would require more elaboration. BUG=none TEST=presubmit passess Change-Id: I4e48d94c9b430d8b224e70e9267abe3795bff7cc Reviewed-on: https://chromium-review.googlesource.com/1719770 Tested-by: Denis Nikitin <denik@chromium.org> Commit-Ready: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-07-25toolchain-utils: add block list for histograms outputJian Cai
Histograms of some tests such as loading.desktop are very different from their chart-json counterpart. It seems it is almost impossible to parse these tests without case-by-case parsing. Parse chart-json output of these tests instead. BUG=chromium:984713 TEST=local tests Change-Id: I225567a1be9d401b5dd0c1f682184e581ac4acce Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1714125 Tested-by: Jian Cai <jiancai@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-07-25Toolchain-utils: update crosbolt listJian Cai
Move loading.desktop and rendering.desktop from crosbolt_perf_tests to telemetry_crosbolt_perf_tests. Otherwise test_that does not pass correct suite type. BUG=chromium:970061 TEST=local tests Change-Id: I485335fd285d0e132e0d7a2fe309a6303931de56 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1717127 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Jian Cai <jiancai@google.com>
2019-07-25Toolchain-utils: replace xrange with range on results_cache.pyJian Cai
xrange builtint is replaced by range on Python3. Use range instead of xrange on results_cache.py. BUG=chromium:986617 TEST=local tests Change-Id: I729b12a92b75dbd355fad3012205765a648be538 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1717449 Tested-by: Jian Cai <jiancai@google.com> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-07-25toolchain-utils: remove all xrangesGeorge Burgess IV
This removes all mention of xrange from toolchain-utils (modulo ones being changed in other CLs that are in flight). It's now an apparent lint error to use xrange, and it hinders our move to python3. As commented on If90d26664c70ccb73750f17573b89933fdb048f4, xrange -> range in python2 is really only a space concern (or speed in pathological cases), so migrations of this nature are generally super straightforward. I glanced at each of these callsites, and none of them appear to be pathological, so my hope is that this should all be Just Fine :) (Also fun to note that this includes a .diff file that has python code embedded in it.) BUG=None TEST=Presubmit tests Change-Id: Ic9f3ac3a5044d7a07da8a249bc505278d98203de Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1717130 Commit-Queue: George Burgess <gbiv@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: George Burgess <gbiv@chromium.org>
2019-07-23toolchain-utils: update crosbolt benchmarksJian Cai
Current crosbolt benchmarks we run are out-of-date. Update the list to keep up with go/crosbolt. BUG=chromium:970061 TEST=localtests Change-Id: Ib572a68257a23cc32d7211bfbfbf4cebfd3f89c0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1700150 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Jian Cai <jiancai@google.com>
2019-07-22crosperf: Init workflow to use skylab to run testsZhizhou Yang
This patch provides init workflow to use skylab to run both original test_that test (aka. client test) and telemetry_Crosperf test. This patch also removes telemetry_run since no one uses it any more. This patch will: 1) Launch skylab to create test to run on specific remote(s). 2) Wait until test finishes, and download results from gs location. 3) Skipping afelock system for lab machines. This patch is missing a lot of important concepts for our final goal, marking them as TODO in the patch. TEST=passed all unittest. BUG=chromium:984790 Change-Id: I7d0d15e67e4610d42ff84aa3fe2f1f1ac90fc1bb Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1707928 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-07-22crosperf: introduce skylab field and pass to experiment and labelZhizhou Yang
This is the initial patch to migrate crosperf from autotest to skylab for lab runs. This patch introduces skylab field in experiment, parsed it in settings_factory and restructured label class and experiment class so that further steps can use skylab field. TEST=Passed all unit tests BUG=chromium:984790 Change-Id: I1104fea739977c522938790012f4f3f522f9d65f Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1705234 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-07-22crosperf: Remove unsupported machines.Manoj Gupta
Remove machines that are now unsupported. BUG=None TEST=None Change-Id: Id8173260ab95f211021f0503ad8d6748af869213 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1712452 Commit-Queue: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Auto-Submit: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-07-10crosperf: Fix a typo in kernel kallsyms.Manoj Gupta
Replace a typo kernal by kernel. This otherwise breaks kernel cycles profiling. BUG=None TEST=unit tests pass Change-Id: I7cee1bd058c256262ca2758273a7309779a7a3c1 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1696566 Commit-Queue: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2019-07-10toolchain-utils:fix a typoJian Cai
BUG=chromium:967868 TEST=local tests Change-Id: I988a30fa0e116eb2df9c17b7947bc5680d19c0fb Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1696384 Reviewed-by: Denis Nikitin <denik@chromium.org> Tested-by: Jian Cai <jiancai@google.com>
2019-07-10crosperf: Recover intel_pstate performance setupDenis Nikitin
Revert back performance governor intel_pstate setup. Change the order of no_turbo setup which should follow the intel_pstate governor setup. BUG=chromium:966514 TEST=verified on samus, eve, rammus, kefka. Change-Id: I40e6930bc62151cafe1a95f7f9d05c138e6d7663 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1687384 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-07-09toolchain-utils: migrate Telemetry tests results from chartjson to histogramsJian Cai
Shift to histograms as charjson format is being deprected for Telemtry tests BUG=chromium:967868 TEST=Local tests. Change-Id: I0645c6f10a93a454cc50090d2b790c9f386d9358 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1691318 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Jian Cai <jiancai@google.com>
2019-05-28crosperf: Stop setting cpu governor.Manoj Gupta
Performance governor makes data noisy for high-end machines while powersave makes speedometer2 timeout. Let's try with the defaults for now and decide a better strategy later. BUG=chromium:966514 TEST=unit tests pass. Change-Id: I12e0f1ce7a0eed52f07cd3e1272d4280b93ef3b0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1632422 Tested-by: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org> Commit-Queue: Manoj Gupta <manojgupta@chromium.org>
2019-05-24crosperf: Set governor to powersave.Manoj Gupta
This is to improve stability in performance data. BUG=chromium:966514 TEST=powersave governor works. Change-Id: I0bb0f8349c69437f154bcdac29fa64d2664fb6ef Reviewed-on: https://chromium-review.googlesource.com/1627301 Commit-Ready: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-05-22crosperf: Add check core online in PinGovernerFrequency().Tiancong Wang
Due to Hyper-Threading is disabled, not all cpus in /sys/devices/system/cpu/cpu* are online on the x86 DUTs. Our PinGovernerFrequency() functions need to write performance to scaling_governor and it fails if the cpu is offline. This patches adds checks before writing to make sure the core is not offline. BUG=None TEST=Local tests. Change-Id: I5dee6851848c18ff9caac0fb5bf5a6a18741a3c8 Reviewed-on: https://chromium-review.googlesource.com/1618095 Commit-Ready: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-04-25crosperf: Use --fast to prevent copying sysinfo for telemetry_CrosperfZhizhou Yang
We don't need sysinfo collected by perf from DUT. --fast will stop copying sysinfo but was orignially disabled for profiling runs because of a conflict in autotest profiler. Now that we don't use autotest profiler any more, this patch will turn on --fast for telemetry_Crosperf no matter profiler_args are present or not. BUG=chromium:956174 TEST=Passed unit tests; Tested both with profiler_args and without, checked the sysinfo directory and it was empty. Change-Id: Id99db48d6ae9043309c5113d0a6d9afd6e3c095b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1583018 Reviewed-by: Luis Lozano <llozano@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-04-18crosperf: generate perf report with correct debug filesZhizhou Yang
This patch fixes the issue in chromium:946588. This patch makes perf report no longer use hard code debug directories. There are several different situations: 1) When running tests on a downloaded image, it will download debug.tgz from gs, extract it to debug_files in /tmp. Options --symfs and --vmlinux will depend on this directory, and throw a warning to user that --kallsyms cannot be applied. 2) If running with downloaded image and debug.tgz could not work, then we will try to use local build, but give user a warning that it may not match real symbols well. 3) When running tests with local build, try to find debug info from /build/$board directory. Thus, this patch added a new field in label, called 'debug_path', if this is manually set in experiment file, then crosperf will directly use the location. Downloading of debug.tgz will only happen when perf_args is set in global settings. TEST=Passed all unit tests, tested with eve and sand. BUG=chromium:946588 Change-Id: I7f35d1216d912c8526d5501748f951face1273aa Reviewed-on: https://chromium-review.googlesource.com/1561780 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-04-08Fix 'result' variable referenced before assignment issueZhizhou Yang
This issue is caused by lacking of coverage in code. When we are parsing json result, value/values doesn't exist in its filed_dict for a key, this issue will be hitten. TEST=Eliminated the error caused by this issue. BUG=chromium:929519 Change-Id: I81e4c0cf5f92c2dca9662ba86629355b3b048089 Reviewed-on: https://chromium-review.googlesource.com/1553681 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-04-05crosperf: make run_tests use our new test runnerGeorge Burgess IV
This is three small changes in one. The main attraction is no functional change. The benefits we see are: A. less cluttered output B. total wall time of test running is down from 18s to 13s on my machine, since the test runner spawns all tests in parallel The second change is that we now set PYTHONPATH for linting. Otherwise, we'll see errors about bad imports. Finally, this fixes a test. It appears that the parameters of this assertRaises function have been accidentally flipped, though that works without issue outside of the chroot. To understand why, consider that assertRaises has two forms that act completely differently: self.assertRaises(Exception, foo) Which causes self.assertRaises to run foo(), and expect that foo() raises an Exception. Additionally, it can be used like so: with self.assertRaises(Exception): foo() Which causes assertRaises to intercept any Exception that the block under it may throw. The difference between the in-chroot and out-of-chroot Python is simple: outside of the chroot, the default value for the second arg to assertRaises -- not counting `self` -- is None. Inside of the chroot, they have a special, hidden sentinel value that it defaults to. So, *outside* of the chroot, `self.assertRaises(Exception)` is equivalent to `self.assertRaises(Exception, None)`, so the latter returns a context object that tries to catch an exception. Inside of the chroot, `self.assertRaises(Exception, None)` is not equivalent to `self.assertRaises(Exception)`, and `None` is called as though it was a function. Since the removed code was `self.assertRaises(foo, None)`, outside of the chroot, we'd just be returned a context object that we dropped on the floor. Inside of the chroot, `self.assertRaises(foo, None)` would try to call `None()`, fail, then try to say `issubclass(exception_that_calling_none_raised, foo)`, but foo is a function, so `issubclass` would raise on its own. For those keeping score at home, the chroot's behavior was introduced in upstream Python in 7f71e04cb510c24be337a22350324dc8a28e9775, which landed in the 2.7.10 release. It wasn't until 2.7.11 that the revert 049060c249a83b69c4841ed37b7f4303f9ad7dd7 took effect. For more, the bug was https://bugs.python.org/issue24134 BUG=None TEST=./run_tests.sh Change-Id: I58c398caafde03242ed3ca7530bf9a819fae99e2 Reviewed-on: https://chromium-review.googlesource.com/1548399 Commit-Ready: George Burgess <gbiv@chromium.org> Tested-by: George Burgess <gbiv@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2019-03-22crosperf: Include score into the result of speedometer2.Tiancong Wang
Add RunsPerMinute__summary to the result of speedometer2. Now it has both total execution time and overall score. BUG=None TEST=Tested locally. Change-Id: I9f20153a40ac965e0fcf55e979b52e5ab53d07cd Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1534039 Reviewed-by: Luis Lozano <llozano@chromium.org> Tested-by: Tiancong Wang <tcwang@google.com> Commit-Queue: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
2019-03-19crosperf: Fix the name of amean column for weighted resultsZhizhou Yang
This fixes the naming of amean column for cwp mode, should be "Weighted Samples Amean" other than just "Amean". BUG=None TEST=Tested with both cwp and general mode. Change-Id: I66c0cce41da4aab2cfc65e867c0e0335e32029ad Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1529231 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2019-03-15crosperf: Feature to ignore min and max value in resultsZhizhou Yang
This patch provides an option for user to ignore min and max value in the results when generating report. User can use this feature by specifying `ignore_min_max: True` (which by default is False) in experiment file. When values count in a single test is smaller than 3, we automatically ignore this option for it, and print out a warning. BUG=chromium:938758 TEST=Tested with examples in general/cwp mode report generation. Passed all unit tests. Change-Id: I36a4c4d99836c201cdd2f2f9f2a4b1a4ffdaa47d Reviewed-on: https://chromium-review.googlesource.com/1521054 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2019-03-14crosperf: Add unittest for disable ASLR CLTiancong Wang
In crrev.com/c/1398522, we disabled ASLR in crosperf by default. In that CL, there's no unittest support and other crosperf unittest might fail because of the CL. This patch is to fix the errors in unittest and also add unittests for the newly added functionalities. BUG=chromium:930332 TEST=Unittests pass Change-Id: I332604245ea2ee1100a498ce021530572e7609ba Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1497036 Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Tiancong Wang <tcwang@google.com>
2019-03-08Swap READMEs to markdownGeorge Burgess IV
Suggested by vapier@ in I7daf1cbe93f8809e469698d7772a6c68baf9e194 This ignores README.chromium, since I'm unsure if that's a special name that some tool somewhere relies on. I skimmed over all of these in gitiles (as you can probably tell by the 35 patch sets :) ) and fixed the issues that jumped out at me. I'm sure that some remain somewhere, but I think I got the majority of them. An attempt has been made to make all files roughly follow the rules here: https://github.com/google/styleguide/blob/gh-pages/docguide/style.md Note that not all of them have super meaningful introductions with ToCs/etc, but the rest should be relatively consistent with that guide. BUG=None TEST=Manual inspection Change-Id: I158048d972d49966e3b8ff1e07e5429928edb339 Reviewed-on: https://chromium-review.googlesource.com/1504893 Commit-Ready: George Burgess <gbiv@chromium.org> Tested-by: George Burgess <gbiv@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-03-07crosperf: do not put key 'samples' in result.keyval when test failedZhizhou Yang
This patch fixed two issues: 1) When determining how many iterations pass/fail, we originally check if samples exists or not in result.keyval. So it always comes up with "all pass" even if test fails and samples equals to 0. 2) Stddev will become large when we include 0s in samples. After this change, the value in failed cell with be 'None', and not be calculated in stddev and avg. BUG=chromium:938582 TEST=Tested with runs with failures, for cwp and general mode. Passed all unittests. Change-Id: If14eff40631be1080f6713f2bfb631843cb2ab62 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1504632 Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-03-06crosperf: enabling result caching for cwp modeZhizhou Yang
This patch provides support for result caching when cwp_dso is specified. This patch introduced cwp_dso into ResultsCache class, so that cwp_dso info will be stored while caching. When populating from cache, samples will be re-calculated with the perf.data file that has been cached before. This patch also: 1) Fixed return value of GetSamples(). 2) Not need to pass weight to result. 3) Added unit tests. BUG=chromium:905854 TEST=Passed all unit tests; Tested with --rerun=True/False, both on cwp and general mode, to make sure cache works. Change-Id: I297fa0aaad781b4a665a4a29bb6fdb4bca0d790b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1501458 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Zhizhou Yang <zhizhouy@google.com>
2019-03-06crosperf: replace cpu cycles in report with samplesZhizhou Yang
What we collected from benchmark run actually are samples from perf tool, so "cpu cycles" is not a accurate name. BUG=chromium:936573 TEST=Tested with cwp and general mode on eve; Passed all unittests. Change-Id: I35533cea0987c4e1b112498cc1b0271eaab665ae Reviewed-on: https://chromium-review.googlesource.com/1495963 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2019-02-12crosperf: Disable ASLR in crosperf by defaultTiancong Wang
Disabling Address Space Layout Randomization (ASLR) can reduce much variance when measuring performance of benchmark from run to run. This CL will disable ASLR in crosperf by default and add a flag '--enable_aslr' for users to enable it for other testing purpose. BUG=chromium:930332 TEST=Tested locally for several boards and several benchmarks Change-Id: Icfbe477a99ef9a68b503e4979cf2ff47d07a128f Reviewed-on: https://chromium-review.googlesource.com/1398522 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Tiancong Wang <tcwang@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-02-08crosperf: in cwp mode, first check whether iterations are the sameZhizhou Yang
In cwp mode, we ask user to set same iterations for different benchmark runs, but if user did not do so, he will spend too long running all benchmarks and then get an error message in the end. This patch checks whether iterations in experiment file are the same at the time we construct benchmark. Also the patch fixed a small test case mismatch in weight check so that all unit tests can pass. Fixed format issues from pylint. BUG=chromium:929390 TEST=All unit tests passed. Change-Id: I0eae51205e8c0fa80131c5b79c4773e88c5b0915 Reviewed-on: https://chromium-review.googlesource.com/1459854 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-11 14:03:52 +0000