aboutsummaryrefslogtreecommitdiff
path: root/crosperf/experiment_factory.py
AgeCommit message (Collapse)Author
2015-12-08crosperf: fix a typo.Yunlian Jiang
BUG=None TEST=the typo is fixed. Change-Id: I11249c57e9483a8d1c1a2ca72853b6870949364d Reviewed-on: https://chrome-internal-review.googlesource.com/240878 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-12-08Fix parameters passed to non-telemetry autotest benchmarks.Caroline Tice
The parameters being passed to the non-telemetry Benchmarks were not quite right. This CL fixes that. BUG=chromium:567333 TEST=Tested in my chroot with bvt tests. Change-Id: Ied834e0904eec7801118d4996045d96c1fb4d329 Reviewed-on: https://chrome-internal-review.googlesource.com/240576 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-11-05Temporarily remove graphics_WebGLAquarium from buildbot testing.Luis Lozano
For the images that we build in trybots, this test cannot be executed because we are not downloading the necessary artifacts for autotest. See chromium:551710. This change temporarily removes that test from the nightly testing that we do based on trybot images. These changes should be removed once this bug is fixed. BUG=chromium:551710 TEST=None Change-Id: I15ec9386d52809492345d9cd46a6f0ad39330dbc Reviewed-on: https://chrome-internal-review.googlesource.com/237838 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-10-12crosperf: remove dhtml from pagecyler test.Yunlian Jiang
Telemetry removed dhtml, so crosperf should remove it accordingly. BUG=none TEST=the benchmark is removed. Change-Id: I0bcab62e9e726d87a31febcc80121086a87337fa Reviewed-on: https://chrome-internal-review.googlesource.com/234195 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-09-25Add compiler info to experiment data.Caroline Tice
Check the build label to see which compiler was used. Store the compiler in the label, and add it to the json filename and output. Also update nightly buildbot script to add 'compiler' field to labels. BUG=None TEST=Tested in my directory. Change-Id: I124c4910cce1145708bb2a52e3a6d6dff93777c3 Reviewed-on: https://chrome-internal-review.googlesource.com/231811 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-08-20Crosperf schedv2 (1) - new option and integrating new scheduler.Han Shen
This Cl introduces a new option '--schedv2' which uses the new scheduler to allocate jobs (benchmark_runs) to duts. With this option, schedv2 takes control of storing/allocating jobs and reimage machines using the new algorithm. This CL leaves actual reimaging and running jobs non-op (a random time sleep is used for each such op, which would be replace in later CLs.) You may try this CL like this and see how schedv2 works - crosperf --locks_dir=/usr/local/google/home/shenhan/tmp --use_file_locks=True --logging_level=verbose --schedv2 some.exp Change-Id: If5bb7751b466c39e54c93fe8f0b4e363be6d9165 Reviewed-on: https://chrome-internal-review.googlesource.com/225515 Commit-Queue: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-07-16Add 'run_local' to run the test harness locally.Ting-Yuan Huang
The option 'run_local: True|False' will be passed into autotest. For example, this is how test_that will be invoked: test_that --arges="run_local=True ..." ... BUG=None TEST=Tested with the folllowing combinations. benchmark: sunspider, smoothness.top_25_smooth perf_args: record -e cycles,instructions / (none) Change-Id: I7315027a7c9433d17a9f3fe54d7e8c3f480ea4f4 Reviewed-on: https://chrome-internal-review.googlesource.com/217370 Reviewed-by: Caroline Tice <cmtice@google.com> Tested-by: Ting-Yuan Huang <laszio@google.com> Commit-Queue: Ting-Yuan Huang <laszio@google.com>
2015-06-30Update all_toolchain_perf to use new recommended set.cmtice
Update the all_toolchain_perf alias to use the new recommended set of tests. Also update the default test values to include default values for the newly recommended tests. Also update the nightly tests to use the newly recommended set. BUG=None TEST=Tested new alias with crosperf. Change-Id: I603f866723646a2d5b6d86fbe1de3c4617e09046 Reviewed-on: https://chrome-internal-review.googlesource.com/220385 Reviewed-by: Han Shen <shenhan@google.com> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com>
2015-06-13Add flag to crosperf to use file lock mechanism.cmtice
This CL adds a flag to crosperf that specifies using the old file locking mechanism, rather than the new AFE server locking mechanism. This is useful when there are issues with accessing the AFE server. BUG=None TEST=Tested crosperf with this, both using the AFE locking mechanism and using the file locking mechanism. Change-Id: If6aa3e97a77d8ccb53f0e8f0405cc7f45e983b75 Reviewed-on: https://chrome-internal-review.googlesource.com/219096 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-05-28Added benchmark retry support.Luis Lozano
We often get spurious failures from the benchmarks runs. This CL adds support for a "retries" field that will automatically retry the benchmark run by the amount specified. BUG=None TEST=Test by hand. Change-Id: Ib881c40d766e67a7daf855ce9137d3e3d631f12e Reviewed-on: https://chrome-internal-review.googlesource.com/217525 Reviewed-by: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org>
2015-04-26Fix current problem with machine locks.cmtice
Currently if crosperf can't access the default locks directory, it assumes that it is safe to proceed without locks and it uses machines without checking locks. This script does two things: It causes crosperf to fail if it can't access a locks directory; and it allows the user to specify an alternate locks directory. As a by-product it also adds a boolean prompting function to utils/misc.py BUG=None TEST=Tested with default directory accessible & inaccessible; tested using local locks directory. Change-Id: I6f24f772986813bf089f5f4e40335a15e8ee4837 Reviewed-on: https://chrome-internal-review.googlesource.com/214129 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-03-26Remove deprecated Telemetry benchmarks.Luis Lozano
A bunch of benchmarks listed in our suites have been removed from Telemetry. Stop trying to use them. BUG=None TEST=Verified by hand Change-Id: Ic624c1867f6129650f9a02f7325d94d7abb970ee Reviewed-on: https://chrome-internal-review.googlesource.com/210560 Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-02-06Make crosperf fail if 'build' field is empty.cmtice
Change-Id: Ia929969e0e82cc16f2ebd7c020b1af7e0b5c84ff Reviewed-on: https://chrome-internal-review.googlesource.com/196107 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-10-22Fix issue where crosperf can't find local cros_scratch.cmtice
Crosperf was assuming the user always has an NFS home directory, and was looking for the cros_scratch directory there. This makes crosperf use the user's home directory, whether it's NFS or local. Also change --share_users, which also assumed NFS home directories, to --share_cache, which requires specification of the cache path(s) to use. BUG=None TEST=Tested this with my new local homedir. Change-Id: I28d15467dfbd25b8b237cefa2979d3fd9464a7a7 Reviewed-on: https://chrome-internal-review.googlesource.com/179978 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-06-02Add --cache_only global setting to crosperf.cmtice
The CL adds a new setting, --cache_only, which causes crosperf to ONLY report test results that it finds in the cache (it will NOT rerun any tests, even if no cache results are found). This is partly for implementing a weekly summary report, where we want to compare the nightly results from the past week, but not actually rerun anything. BUG=None TEST=I tested this option while generating weekly test reports, and it worked as it is supposed to. Change-Id: Icb1f5343f76ce999bebd0810bb86ab64a7577313 Reviewed-on: https://chrome-internal-review.googlesource.com/164890 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-05-12Expand experiment_factory_unittest.py to be more complete.cmtice
This CL adds more testing to experiment_factory_unittest.py, and it fixes one small bug in experiment_factory.py. BUG=None TEST=Ran unittest. Change-Id: I9fd13c05d793bb71accf3e19d361dff79f7110e9 Reviewed-on: https://chrome-internal-review.googlesource.com/163149 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-21Clean up Crosperf options and help.cmtice
Crosperf's help was completely out of date and incorrect. Fixed that. Also, I cleaned up the options, removed some that were no longer appropriate or not ever used, and made some options global-only. Also removed all references to pyauto. We no longer use or run pyauto tests. Option changes: Completely removed outlier_range (it was not actually doing anything), key_results_only (applied only to pyauto tests), use_test_that (not needed now that we're not using pyauto) and md5sum (never used). Made the following options global-only: rm_chroot_tmp, perf_args, logging_level, show_all_results, board. Removed 'build' from global settings and made it an image-only setting (it made no sense as a global setting). I also adjusted the unittests to work with these changes. BUG=None TEST=I tested all the options in all their various positions; I tested with and wtihout caching; and I tested local and official images. I also ran the unittests. Everything works. Change-Id: I7653663f65c0a3363a1904c0220f1066e72c7f9d Reviewed-on: https://chrome-internal-review.googlesource.com/160758 Reviewed-by: Bhaskar Janakiraman <bjanakiraman@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-12Download official & trybot images; treat as local images for tests.cmtice
Currently we cannot get md5sums for official & trybot images, which causes multiple problems: Crosperf re-images the DUT every time it runs a test for an image, even if the image is already on the DUT; if crosperf is running an experiment with multiple official or trybot images, it can get confused about which results belong to which image; caching does not work properly without md5sums. To fix all of these problems, this CL changes Crosperf to download the official or trybot images into the /tmp directory in the chroot, and from there it can treat them as local images (getting md5sums etc). In order to download them, it first has to translate the xbuddy syntax (which can contain aliases such as 'lumpy/dev-latest') into the actual image name (e.g. lumpy-release/R36-5727.0.0). This translation is done by a new script, translate_xbuddy.py, which must be run from the toolchain-utils directory inside the chroot. BUG=356279,356474,356476 TEST=Tested with official and trybot images, using both correct designations and xbuddy aliases; tested with the image missing from the chroot and with it already in /tmp; tested with and without caching. Also tested with the experiment file included in the 3 bugs. Change-Id: I917b5520604b7d4851db2c8123165e81b866da2b Reviewed-on: https://chrome-internal-review.googlesource.com/159465 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-04-03Add page_cycler.bloat to Toolchain team recommended test set.cmtice
BUG=None TEST=Ran test with this change. Change-Id: I89aab58c8890a7c3f36707d2ffe7422fbfa17547 Reviewed-on: https://chrome-internal-review.googlesource.com/158585 Reviewed-by: Yunlian Jiang <yunlian@google.com> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com>
2014-03-19Various small fixes for nightly infrastructure tests.cmtice
- Change the number of iterations for tests from 1 to 3. - Increase the timeout for image_chromeos.py, so 'cros flash', which takes a bit longer than cros_image_to_target.sh did, doesn't get killed just as its finishing. - Comment out smoothness.top_25 for now, since it's broken and/or hanging. BUG=None TEST=Successfully ran nightly infrastructure tests with these changes. Change-Id: I77b663b1910fe12ef927afdf15d88a7c1e82d491 Reviewed-on: https://chrome-internal-review.googlesource.com/157419 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-03-14Add --logging_level option to crosperf.cmtice
Add --logging_level={verbose, average, quiet} option to crosperf, optionally reducing the amount of logging output. "verbose" gives the same output Crosperf always gave; "quiet" gives minimal output, and "average" is in the middle. The default for now is "verbose". Doing this also involved adding logging levels to the command_executer, and to image_chromeos (again, with the default being "verbose"). BUG=None TEST=Tested all three levels of logging, with and without having to reimage the DUT. It all worked properly. Change-Id: Icf8c9a6831fe25202adcb624c6c7e1d1a7ac25a5 Reviewed-on: https://chrome-internal-review.googlesource.com/156883 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-02-20Add 'build' settings option and xbuddy image format.cmtice
This patch updates Crosperf to add the 'build' tag in the experiment file, and to allow that field to contain xbuddy syntax for using trybot and/or official builds in the test runs. It also adds a bit more checking to make sure we have everything necessary for running 'cros flash' before attempting to use it. BUG=None TEST=I have run this using an experiment file that compares a local image, a trybot image and an official image against each other. It all worked. Change-Id: Ia896799061508fb5718a3201b1599d8622de0b3f Reviewed-on: https://chrome-internal-review.googlesource.com/154097 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-01-15Add test group containing recommended Telemetry performance testscmtice
for toolchain group. Change-Id: I178975045ed78c93800d4d6c1f6c86fcb88e3ecd Reviewed-on: https://chrome-internal-review.googlesource.com/151653 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-12-10Use default results for Telemetry tests; add option to use all results.cmtice
When running telemetry_Crosperf, use the json file of report defaults to determine which results to actually include in the report. This CL also adds a flag to Crospref, --show_all_results=True, that allows the user to override the defaults and generate a report showing all the values. BUG=None TEST=Tested this both with and without the --show_all_results flag. It works as expected for me. Change-Id: I91364572ebdf3d54bb8b408fbd5a2580923f6829 Reviewed-on: https://chrome-internal-review.googlesource.com/149457 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-11-06Make changes to run Telemetry tests in nightly infrastructure tests.cmtice
BUG=None TEST=Tested this running the nightly infrastructure cron jobs. Change-Id: I6cccee222b074e9410b45df6750c097430207746 Reviewed-on: https://chrome-internal-review.googlesource.com/147515 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-10-23Fixed name of telemetry benchmark page_cycler.basic_insertLuis Lozano
BUG=None TEST=Tested by hand using the new name Change-Id: I7246f9f5d2a5e17133613bc4f9038fe1b42332e9 Reviewed-on: https://chrome-internal-review.googlesource.com/146909 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org>
2013-10-02Allow "." in benchmark names; Allow multiple test sets.Caroline Tice
This fixes a few minor problems with crosperf: 1). When using run_remote_tests.sh, it adds the "--use_emerged" flag, which is a recent requirement. 2). It updates the regular expression parsing to not mess up escaped characters, and to allow "." in benchmark names (common in Telemetry). 3). It allows there to be multiple "sets" of benchmarks that can be specified together, e.g. perfv2 or pagecyclers (or more in the future). BUG=None TEST=I've run all the various telemetry_Crosperf variations (individual perf_v2 tests; individual telemetry page cycler tests; 'all_perf v2', and 'all_pagecyclers'. Change-Id: Idd45caa6487780b92b8eb321b37b873978f96dca Reviewed-on: https://chrome-internal-review.googlesource.com/145305 Reviewed-by: Yunlian Jiang <yunlian@google.com> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com>
2013-09-23Update crosperf to run telemetry test via autotest.Caroline Tice
This updates crosperf to recognize the suite 'telemetry_Crosperf', which allows users to run telemetry performance tests via test_that and the telemetry_Crosperf autotest. This won't be usable in general until CL 168491 (the telemetry_Crosperf CL) gets committed. BUG=None TEST=Invoked telemetry tests using this change. Note: Profilers don't quite work properly with telemetry_Crosperf yet, but everything else does. Change-Id: Id14edc89db79bc244496e8f6a5b614300a00be54 Reviewed-on: https://chrome-internal-review.googlesource.com/144440 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-08-22Make test_that available for running tests.Caroline Tice
Add a new flag, --use_test_that=True/False, to crosperf, to allow users to tell crosperf to use the test_that script rather than the run_remote_tests.sh script for running their tests. BUG=None TEST=Tested with and without the flag, to verify that it does the right thing. Change-Id: I1a040d23adbf364b4a47320d9fd759a2d46e6689 Reviewed-on: https://gerrit-int.chromium.org/43109 Reviewed-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-07-01crosperf: filter out "/' in remoteYunlian Jiang
TEST=nightly test passes. BUG=None Change-Id: I525e488c9f4c47cfe09bd8c15d06997b9e75f140 Reviewed-on: https://gerrit-int.chromium.org/40463 Reviewed-by: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Yunlian Jiang <yunlian@google.com>
2013-06-06Adding support of telemetry to crosperfYunlian Jiang
BUG=None TEST=run crosperf with pyauto/telemetry test with/without cache. all pass. Change-Id: If07ac020a9107a79d5780a58fd6dcc924d07f07f Reviewed-on: https://gerrit-int.chromium.org/36594 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-on: https://gerrit-int.chromium.org/39241 Reviewed-by: Yunlian Jiang <yunlian@google.com>
2013-03-28Add option to specify output dir of crosperfYunlian Jiang
BUG=None TEST=run crosperf with this option, the results are in the right dir Change-Id: Id94251c72297f18f275564a1fd1ef0972227ddfc Reviewed-on: https://gerrit-int.chromium.org/34621 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com>
2013-03-15Cleaned up directory after copy of tools from perforce directoryLuis Lozano
Got rid of stale copies of some tools like "crosperf" and moved all files under v14 directory (that came from perforce) into the top directory. BUG=None TEST=None Change-Id: I408d17a36ceb00e74db71403d2351fd466a14f8e Reviewed-on: https://gerrit-int.chromium.org/33887 Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org>
2012-12-20Synced repos to: 64740Ahmad Sharif
2012-10-09Synced repos to: 63271Ahmad Sharif
2012-02-02Synced repos to: 58208Ahmad Sharif
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-11 06:20:45 +0000