aboutsummaryrefslogtreecommitdiff
path: root/crosperf/results_organizer.py
AgeCommit message (Collapse)Author
2020-01-30crosperf: migration to python 3Zhizhou Yang
This patch migrates crosperf and its utils to python 3. TEST=Passed presubmit check; tested with simple experiment locally. BUG=chromium:1011676 Change-Id: Ib2a9f9c7cf6a1bb1d0b42a1dd3d9e3cbb4d70a36 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2003796 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-09-26crosperf: Add cpu temp/freq in cwp reportDenis Nikitin
Result organizer with CWP has a different control flow and didn't include cpu frequency and temperature data. Current change adds missing data into the Full table of the report file. BUG=None TEST=tested on veyron with cwp experiment file. Change-Id: I67447ab016cee8265c1937b236cc5c38330576e9 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1827918 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2019-08-19crosperf: Add ARM CPU stats in crosperf reportDenis Nikitin
Check for cpustats.log file in results_dir and extract data with CPU frequncy and temperature. Calculate avg/min/max values from the measurements and add to result keyvals. The parser attempts to read turbostat.log first (which comes from Intel devices along with cpustats.log) and only if it doesn't exist or empty switches to cpustats.log (usually ARM devices). BUG=chromium:966514 TEST=tested locally on veyron, scarlet, kevin64 (arm), eve (intel) Change-Id: I12991b39a0f1acb6197ca663a6876b0f013d2026 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1753164 Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-07-28crosperf: Add cpu statistics into results reportDenis Nikitin
Check for turbostat.log file in results directory and if file exists parse the context and extract measurements of CPU Frequency and temperature for ALL cores (not each). Keys added to keyvals: 'cpufreq_min', 'cpufreq_max', 'cpufreq_avg', 'cputemp'. BUG=chromium:966514 TEST=tested all crosbolt perf benchmarks Change-Id: If90d26664c70ccb73750f17573b89933fdb048f4 Reviewed-on: https://chromium-review.googlesource.com/1709824 Tested-by: Denis Nikitin <denik@chromium.org> Commit-Ready: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2019-07-26crosperf: Fix pylint warningsDenis Nikitin
Pylint throws a bunch of warnings on results_cache.py and results_organizer.py (other source files not checked). The change fixes most warnings EXCEPT docstring which would require more elaboration. BUG=none TEST=presubmit passess Change-Id: I4e48d94c9b430d8b224e70e9267abe3795bff7cc Reviewed-on: https://chromium-review.googlesource.com/1719770 Tested-by: Denis Nikitin <denik@chromium.org> Commit-Ready: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-03-06crosperf: replace cpu cycles in report with samplesZhizhou Yang
What we collected from benchmark run actually are samples from perf tool, so "cpu cycles" is not a accurate name. BUG=chromium:936573 TEST=Tested with cwp and general mode on eve; Passed all unittests. Change-Id: I35533cea0987c4e1b112498cc1b0271eaab665ae Reviewed-on: https://chromium-review.googlesource.com/1495963 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-12-28crosperf: Generate report for cwp approximationZhizhou Yang
TODO: unit tests needed. This patch trys to generate a new type of report when cwp_dso is specified. The report will: 1) Generate a new summary table at benchmark level with weighted cpu cycle data. 2) Do not generate perf table. 3) Generate a full table with only cpu cycles in it. TEST=generated correct report with a normal run and two cwp_dso runs: one succeeded run and one with failures. BUG=chromium:902785 Change-Id: I0c62000c6690c66d50a7a4de3cfc280853b134e2 Reviewed-on: https://chromium-review.googlesource.com/1376571 Commit-Ready: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-12-20toolchain-utils: Add loading.desktop@@typical to nightly tests.Caroline Tice
page_cycler_v2.typical_25 is deprecated (and flaky!). The recommended replacement is loading.desktop@@typical. This CL updates our nightly performance test infrastructure to handle loading.desktop@@typical. BUG=chromium:916340 TEST=Tested with nightly run on chrotomation2. Change-Id: I6366b3a8e6a2926ad759a209cfded890e3109810 Reviewed-on: https://chromium-review.googlesource.com/1382826 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-06-06Crosperf: Mark benchmark run as failed if it contains no results.Manoj Gupta
Sometimes telemetry tests do not fail but results are empty. In that case, mark the run as failed explicitly. BUG=chromium:729634 TEST=Tested manually on failing lummpy nightly tests. The generated report shows tests as failed as expected. Change-Id: Ia52af887e81ed630d7578f9abe93f9656c70512b Reviewed-on: https://chromium-review.googlesource.com/526392 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Manoj Gupta <manojgupta@chromium.org> Trybot-Ready: Manoj Gupta <manojgupta@chromium.org>
2017-04-08[crosperf] Fix status reporting for multiple iters inside a single run.Caroline Tice
If a single benchmark run contains multiple iterations (like BootPerfServer), then only the first internal iteration was getting the correct pass/fail status, so even though everything passed the report title would claim 1 pass and 9 fails (for 10 internal iterations). This fixes that. BUG=chromium:426960 TEST=Ran BootPerfServer with change. Now shows correct number of passes/fails for the iterations. Change-Id: I9638b39c94692fd754698e64d89cb22c10e9d479 Reviewed-on: https://chromium-review.googlesource.com/470547 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@chromium.org>
2017-04-08[toolchain-utils] Fix remaining lint errors in toolchain-utils.Caroline Tice
In addition to fixing the lint errors, this also fixes the Python formatting issues (ran tc_pyformat on nearly all the files). BUG=chromium:570450 TEST=Ran all crosperf & bisect tool unit tests. Ran afe_lock_machine.py (check machine status) Ran full crosperf test (octane, speedometer, BootPerf) on alex. Change-Id: Ic86f9192801ac67769f3de30f1c5f0d203ce0831 Reviewed-on: https://chromium-review.googlesource.com/471886 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2016-08-26crosperf: refactor results_organizer; NFC.George Burgess IV
As part of playing with results_report (so I could make it less ChromeOS-dependent), I started refactoring results_organizer, which is only used by results_report. This was the result. The goal was to make it easier to understand what, exactly, results_organizer is trying to do. BUG=chromium:641098 TEST=./test_all.sh passes. Change-Id: I8e53175fbc1ded6f23631550223905b3321376ca Reviewed-on: https://chrome-internal-review.googlesource.com/280815 Commit-Ready: George Burgess <gbiv@google.com> Tested-by: George Burgess <gbiv@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-10crosperf: clean results_organizer.pyYunlian Jiang
BUG=chromium:567921 TEST=it is lint clean now. Change-Id: I0cf9445f5db822462c53d75a138e984070319714 Reviewed-on: https://chrome-internal-review.googlesource.com/241429 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-11-04Fixed bad key problem while generating json report.Luis Lozano
In some weird cases, telemetry benchmarks incorrectly return empty performance results. This was causing an invalid key error while generating the json report. BUG=chromium:551105 TEST=Forced generation of empty results and tested we dont have the dictionary missing key issue. Change-Id: I32803c573f1d268e010313292b99c0017a5dff92 Reviewed-on: https://chrome-internal-review.googlesource.com/237516 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-09-10Store machine description info in cache and json reports.Caroline Tice
This CL updates the cache mechanism to store the machine spec info (the checksum string) in the cache, as well as storing the machine information in the json reports (to go into the dashboard archive). It also updates the json reports to always include the full set of result fields for a test, and it verifies that the old cache directory exists before trying to remove it. BUG=None TEST=Tested in my directory, with --cache_only=True, --rerun=True and --json_report=True Change-Id: Ib12bae4ca10141de0b972e3a7aeee83c7b59049b Reviewed-on: https://chrome-internal-review.googlesource.com/229765 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2014-04-21Clean up Crosperf options and help.cmtice
Crosperf's help was completely out of date and incorrect. Fixed that. Also, I cleaned up the options, removed some that were no longer appropriate or not ever used, and made some options global-only. Also removed all references to pyauto. We no longer use or run pyauto tests. Option changes: Completely removed outlier_range (it was not actually doing anything), key_results_only (applied only to pyauto tests), use_test_that (not needed now that we're not using pyauto) and md5sum (never used). Made the following options global-only: rm_chroot_tmp, perf_args, logging_level, show_all_results, board. Removed 'build' from global settings and made it an image-only setting (it made no sense as a global setting). I also adjusted the unittests to work with these changes. BUG=None TEST=I tested all the options in all their various positions; I tested with and wtihout caching; and I tested local and official images. I also ran the unittests. Everything works. Change-Id: I7653663f65c0a3363a1904c0220f1066e72c7f9d Reviewed-on: https://chrome-internal-review.googlesource.com/160758 Reviewed-by: Bhaskar Janakiraman <bjanakiraman@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-04Check for 'significant' results at report generation instead of caching.cmtice
The current implementation of the option that masks unimportant results discards the uninteresting results before caching the test results. After further thought, that was not the right design choice. Among other things it can cause confusing results, such as seen in issue 357346. We should always cache all the results, and do the result filtering during report generation. This CL makes that change. BUG=357346,357343 TEST=Ran crosperf tests with and without cache hits, and with and without entreies in the json file. It all seems to work as expected. Change-Id: I778e5614c73bf751ebaa2d4606af636275247c60 Reviewed-on: https://chrome-internal-review.googlesource.com/159108 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-06-06Adding support of telemetry to crosperfYunlian Jiang
BUG=None TEST=run crosperf with pyauto/telemetry test with/without cache. all pass. Change-Id: If07ac020a9107a79d5780a58fd6dcc924d07f07f Reviewed-on: https://gerrit-int.chromium.org/36594 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-on: https://gerrit-int.chromium.org/39241 Reviewed-by: Yunlian Jiang <yunlian@google.com>
2013-03-15Cleaned up directory after copy of tools from perforce directoryLuis Lozano
Got rid of stale copies of some tools like "crosperf" and moved all files under v14 directory (that came from perforce) into the top directory. BUG=None TEST=None Change-Id: I408d17a36ceb00e74db71403d2351fd466a14f8e Reviewed-on: https://gerrit-int.chromium.org/33887 Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org>
2012-12-20Synced repos to: 64740Ahmad Sharif
2012-10-09Synced repos to: 63271Ahmad Sharif
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-03 02:18:40 +0000