aboutsummaryrefslogtreecommitdiff
path: root/image_chromeos.py
AgeCommit message (Collapse)Author
2022-09-13Update license boilerplate text in source code filesMike Frysinger
Normally we don't do this, but enough changes have accumulated that we're doing a tree-wide one-off update of the name & style. BUG=chromium:1098010 TEST=`repo upload` works Change-Id: Icb42e5012a87920c2cd13b666fb3e55e7e4fb3b8 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/3891080 Auto-Submit: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> Commit-Queue: George Burgess <gbiv@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2022-09-07remove `from __future__ import ...` directivesGeorge Burgess IV
These are only useful when we're running code in a Python 2.7 interpreter. Since we no longer support python2, drop these. BUG=b:244644217 TEST=run_tests_for.py shows no new failures Change-Id: Ief9a12b87a560ab38ca71668636874bcb434a0b3 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/3877339 Reviewed-by: Ryan Beltran <ryanbeltran@chromium.org> Commit-Queue: George Burgess <gbiv@chromium.org> Reviewed-by: Jordan Abrahams-Whitehead <ajordanr@google.com> Tested-by: George Burgess <gbiv@chromium.org>
2022-09-07Autoformat all Python codeGeorge Burgess IV
This autoformats all Python code with our new Python formatter, `black`. BUG=b:244644217 TEST=None Change-Id: I15ee49233d98fb6295c0c53c129bbf8e78e0d9ff Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/3877337 Tested-by: George Burgess <gbiv@chromium.org> Reviewed-by: Jordan Abrahams-Whitehead <ajordanr@google.com> Commit-Queue: George Burgess <gbiv@chromium.org>
2022-04-21toolchain_utils: s/Cr OS/CrOS/gGeorge Burgess IV
Result of running `sed -ri 's/Chrom(ium|e) OS/Chrom\1OS/g' $(find -type f)`. BUG=None TEST=None Change-Id: I59be92537aa19bc989f52b585e307e76dbde401b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/3600147 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: George Burgess <gbiv@chromium.org> Tested-by: George Burgess <gbiv@chromium.org>
2020-01-30crosperf: migration to python 3Zhizhou Yang
This patch migrates crosperf and its utils to python 3. TEST=Passed presubmit check; tested with simple experiment locally. BUG=chromium:1011676 Change-Id: Ib2a9f9c7cf6a1bb1d0b42a1dd3d9e3cbb4d70a36 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2003796 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2019-08-02crosperf: Setup intel_pstateDenis Nikitin
Setup intel_pstate in kernel command line to a value from global settings "intel_pstate" argument (default ''). Added crosperf "--no_hwp" argument for a shortcut change. Kernel command line will be updated only if hwp is supported by CPU. Intel_pstate HWP feature can be disabled via kernel command line argument intel_pstate=no_hwp. See https://www.kernel.org/doc/html/v4.12/admin-guide/pm/ intel_pstate.html#kernel-command-line-options-for-intel-pstate BUG=chromium:966514 TEST=tested on eve, rammus (supported), kefka, samus (not supported), scarlet (arm). Change-Id: Id6b7cf7e230e17788c1024da2aacbc9a16b7ac8e Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1711517 Commit-Queue: Denis Nikitin <denik@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Denis Nikitin <denik@chromium.org>
2019-06-27toolchain-utils: enable crosperf for llvm-toolchain imagesJian Cai
Currently crosperf checks the image it runs on by checking if there is a "test" on the CHROMEOS_RELEASE_DESCRIPTION in the lsb_release file. ChromeOS images built by llvm-toolchain does not have this word, so crosperf always fails to run on such images. This change fixes this issue by checking CHROMEOS_RELEASE_TRACK instead. BUG=chromium:977705 TEST=Local tests. Change-Id: I85025f02c32fcfdc1a3d3e8443350b215ffe83e2 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1672131 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Jian Cai <jiancai@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-04-28crosperf: Reboot dut even if checksums match.Manoj Gupta
We have seen problems where if machine has the image preinstalled but the machine has been ON for a long time, the reports can be bizzare. So lets reboot the machine before the experiments. BUG=chromium:957149 TEST=verified that machine reboots with image preinstalled. Change-Id: I32af330700f9b60801c365cb20ac54b304a82c3a Reviewed-on: https://chromium-review.googlesource.com/1585616 Commit-Ready: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-05-16toolchain-utils: Delete temporary chroot image after verification step.Gabriel Marin
When a local image is outside the chroot, it is temporarily copied into the chroot so that it can be mounted by chroot scripts. Changed the logic to delay the cleanup of the temporary copy until after the verification step. The image needs to be mounted once again during verification. Didn't catch this bug earlier, as the images were cached inside the chroot by previous failed runs. BUG=b:79376118 TEST=Tested with crosperf, two local images and a clean chroot without any cached images. Change-Id: I700f9aa0d9827aa349e75ef7b6b22618baf26fa9 Reviewed-on: https://chromium-review.googlesource.com/1062330 Commit-Ready: Gabriel Marin <gmx@chromium.org> Tested-by: Gabriel Marin <gmx@chromium.org> Reviewed-by: Gabriel Marin <gmx@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@chromium.org>
2018-05-16toolchain-utils: Use the chromeos_image value computed initially.Gabriel Marin
When reimaging with a local image, use the chromeos_image value computed during the initial analysis, instead of attempting to recompute it at each use. The recomputation at the verification stage was also incorrect. BUG=b:79376118 TEST=Tested with crosperf and two local images. The script correctly switches between images and it doesn't fail during the reimaging step. Change-Id: Ie40ed52999ecbebb4e2f5dd3e0aa78a18482533a Reviewed-on: https://chromium-review.googlesource.com/1058181 Commit-Ready: Gabriel Marin <gmx@chromium.org> Tested-by: Gabriel Marin <gmx@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2018-03-08[crosperf] Fix mount and image-finding issues.Caroline Tice
Update MountImage to only call mount_gpt_image.sh from inside the chroot; also update to allow extra commands to be executed on the mounted image. Add FindChromeOSImage to take an external path to a ChromeOS image and convert it to an internal (inside chroot) path. BUG=chromium:820185 TEST=Ran crosperf with these changes on chrotomation2, and it worked. Change-Id: I797b290987e04975fdd62f926838e06e833f18ae Reviewed-on: https://chromium-review.googlesource.com/956238 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-05-17Disable beeps post reboots in crosperf.Manoj Gupta
Run gbb command before flashing to avoid post reboot beeps if possible. It works only on machines with Write Protect Disabled. Crosperf prints the erros messages and continues if this fails. BUG=chromium:697609 TEST=Tested on kevin, daisy and jaq. Change-Id: Ifb0c819281b3215655b7e3b02db605fa073d0e12 Reviewed-on: https://chromium-review.googlesource.com/508153 Commit-Ready: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-04-08[toolchain-utils] Fix remaining lint errors in toolchain-utils.Caroline Tice
In addition to fixing the lint errors, this also fixes the Python formatting issues (ran tc_pyformat on nearly all the files). BUG=chromium:570450 TEST=Ran all crosperf & bisect tool unit tests. Ran afe_lock_machine.py (check machine status) Ran full crosperf test (octane, speedometer, BootPerf) on alex. Change-Id: Ic86f9192801ac67769f3de30f1c5f0d203ce0831 Reviewed-on: https://chromium-review.googlesource.com/471886 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2016-07-27[crosperf] Clean up exceptions.Caroline Tice
Found all instances of 'raise Exception' and changed them to raise something slightly more specific. Changed a few 'except Exception' statements appropriately as well. BUG=chromium:566255 TEST=None Change-Id: If2666bd55838342bc71431e5e6ab2157c2c9e125 Reviewed-on: https://chrome-internal-review.googlesource.com/270731 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-01-28image_chromeos: Rewrite cros flash retry and xbuddy detectionDavid Sharp
Rewrite the retry loop so that the first try is also in the loop. (Reducing duplicated code.) Use str.startswith to search for "xbuddy://". (Say what you mean.) Simplify assembly of cros_flash_args to reduce duplication. TEST=running crosperf BUG=none Change-Id: I6158ce0ce0ad6a5e76492177a6d4700be3912f27 Reviewed-on: https://chrome-internal-review.googlesource.com/245902 Commit-Ready: David Sharp <dhsharp@google.com> Tested-by: David Sharp <dhsharp@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-22Fix cros lint errors.Caroline Tice
Also move deprecated scripts to the 'deprecated' directory. BUG=chromiumos:570464 TEST=tested scripts to make sure they still work. Change-Id: I3442a86d898104591233a0849ea0bafb52ecf1f7 Reviewed-on: https://chrome-internal-review.googlesource.com/244221 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-15crosperf: RunCommand should return one type of object.Luis Lozano
Cleaned up the interfaces for the RunCommand routines. These were returning different types (int or tuple) depending on the value of the return_ouput parameter. Returning different unrelated types from a routine is bad practice. Linter complains about this with several warnings like this: "Attempting to unpack a non-sequence defined at line XY of utils.command_executer" BUG=chromium:566256 TEST=ran crosperf with a example experiment file Ran run_tests. Change-Id: Ibb83ab9322c87558077fc4937ef5c0686bbe5417 Reviewed-on: https://chrome-internal-review.googlesource.com/241459 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Han Shen <shenhan@google.com>
2015-08-20Crosperf schedv2 (1) - new option and integrating new scheduler.Han Shen
This Cl introduces a new option '--schedv2' which uses the new scheduler to allocate jobs (benchmark_runs) to duts. With this option, schedv2 takes control of storing/allocating jobs and reimage machines using the new algorithm. This CL leaves actual reimaging and running jobs non-op (a random time sleep is used for each such op, which would be replace in later CLs.) You may try this CL like this and see how schedv2 works - crosperf --locks_dir=/usr/local/google/home/shenhan/tmp --use_file_locks=True --logging_level=verbose --schedv2 some.exp Change-Id: If5bb7751b466c39e54c93fe8f0b4e363be6d9165 Reviewed-on: https://chrome-internal-review.googlesource.com/225515 Commit-Queue: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-06-24toolchain-utils: specify full path in script to be run in chroot.Rahul Chaudhry
Remove the assumption that the current directory is ~/trunk/src/scripts/ when the chroot is entered with cros_sdk. BUG=chromium:429765 TEST=Ran image_chromeos.py with the changes. Change-Id: I4aa9f233da5b3eef727009d044acdd1a27ea4873 Reviewed-on: https://chrome-internal-review.googlesource.com/219940 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com>
2015-06-10Implement new global locking scheme for machines.cmtice
This CL implements a new machine locking mechanism using Autotest Front End servers. When locking/unlocking a lab machine, it uses the ChromeOS HW lab server; when locking/unlocking a local machine, it uses a local AFE server on chrotomation2. BUG=None TEST=Tested the script manually to lock/unlock machines and query status. Also tested with image_chromeos and with crosperf scripts. Change-Id: I2793bc1f7dc056e725694e81ded656d9f49d227b Reviewed-on: https://chrome-internal-review.googlesource.com/217560 Reviewed-by: Luis Lozano <llozano@chromium.org> Reviewed-by: David Sharp <dhsharp@google.com> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com>
2015-05-21Fix confusing error message when cherrypy is not installed on DUT.Luis Lozano
Also fixed a few warnings. BUG=None TEST=Installed a couple of images on machines with and without cherrypy. Change-Id: I40d3fa1882e3bceb0e70a8d040a892e894cf50db Reviewed-on: https://chrome-internal-review.googlesource.com/217045 Reviewed-by: Caroline Tice <cmtice@google.com> Tested-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org>
2015-03-26Remove checking 'cros flash' locally.Han Shen
The 'cros flash' location is changed recently, and since cros flash should be present everywhere, we just remove the local checking part. However, we keep the part that checks remote's cherrypi module. Change-Id: I3fa06e46eb3cb1d56c03f87ed5f9b632170fabcc Reviewed-on: https://chrome-internal-review.googlesource.com/210270 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com>
2015-03-25Remove invocation to 'cros_image_to_taget.py', which was deprecated on Feb 4.Han Shen
Nightly failed because image_chromeos.py used 'cros_image_to_target.py' (this part still needs to examination ) to image the machines. Change-Id: I918e4a99a301ab6cc120fccd0b01c454597b8336 Reviewed-on: https://chrome-internal-review.googlesource.com/210135 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com>
2014-04-22Use CrosRunCommand to find if cherrypy is available. Fixed bug in CrosRunCommandLuis Lozano
Call CrosRunCommand instead of calling ssh directly to find if cherrypy is available. Also fix bug in CrosRunCommand. The retval returned was always 0. BUG=None TEST=Reproduced with small test cases. Change-Id: I277416b1809eb7287a75bd224b4154f442132a52 Reviewed-on: https://chrome-internal-review.googlesource.com/161123 Reviewed-by: Han Shen <shenhan@google.com> Tested-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com>
2014-04-22Fixed command to figure out if cherrypi is available on the DUTLuis Lozano
Figured out why we sometimes we were not using cros flash for imaging the machines. It turns out the ssh command used was incorrect and (did not specify an empty file for the known hosts) so it will sometimes will fail when connecting to the DUT. There was another problem because the code assumed a return value of non-zero meant cherrypi was not available. We know also check for error code 255 which means the ssh itself failed (not the remote command). BUG=None TEST=Imaged a machine Change-Id: I7c5cef859b73bc9f2a777825914a1c1bb162704b Reviewed-on: https://chrome-internal-review.googlesource.com/161009 Reviewed-by: Han Shen <shenhan@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org>
2014-04-17Do not generate silly ssh warning while checking for cherrypiLuis Lozano
BUG=None TEST=Tested in nightly testing Change-Id: Ia46a0cbb34890a0cb1222f16157c7c98179ec033 Reviewed-on: https://chrome-internal-review.googlesource.com/160769 Reviewed-by: Caroline Tice <cmtice@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org>
2014-04-12Download official & trybot images; treat as local images for tests.cmtice
Currently we cannot get md5sums for official & trybot images, which causes multiple problems: Crosperf re-images the DUT every time it runs a test for an image, even if the image is already on the DUT; if crosperf is running an experiment with multiple official or trybot images, it can get confused about which results belong to which image; caching does not work properly without md5sums. To fix all of these problems, this CL changes Crosperf to download the official or trybot images into the /tmp directory in the chroot, and from there it can treat them as local images (getting md5sums etc). In order to download them, it first has to translate the xbuddy syntax (which can contain aliases such as 'lumpy/dev-latest') into the actual image name (e.g. lumpy-release/R36-5727.0.0). This translation is done by a new script, translate_xbuddy.py, which must be run from the toolchain-utils directory inside the chroot. BUG=356279,356474,356476 TEST=Tested with official and trybot images, using both correct designations and xbuddy aliases; tested with the image missing from the chroot and with it already in /tmp; tested with and without caching. Also tested with the experiment file included in the 3 bugs. Change-Id: I917b5520604b7d4851db2c8123165e81b866da2b Reviewed-on: https://chrome-internal-review.googlesource.com/159465 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-03-19Various small fixes for nightly infrastructure tests.cmtice
- Change the number of iterations for tests from 1 to 3. - Increase the timeout for image_chromeos.py, so 'cros flash', which takes a bit longer than cros_image_to_target.sh did, doesn't get killed just as its finishing. - Comment out smoothness.top_25 for now, since it's broken and/or hanging. BUG=None TEST=Successfully ran nightly infrastructure tests with these changes. Change-Id: I77b663b1910fe12ef927afdf15d88a7c1e82d491 Reviewed-on: https://chrome-internal-review.googlesource.com/157419 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-03-14Add --logging_level option to crosperf.cmtice
Add --logging_level={verbose, average, quiet} option to crosperf, optionally reducing the amount of logging output. "verbose" gives the same output Crosperf always gave; "quiet" gives minimal output, and "average" is in the middle. The default for now is "verbose". Doing this also involved adding logging levels to the command_executer, and to image_chromeos (again, with the default being "verbose"). BUG=None TEST=Tested all three levels of logging, with and without having to reimage the DUT. It all worked properly. Change-Id: Icf8c9a6831fe25202adcb624c6c7e1d1a7ac25a5 Reviewed-on: https://chrome-internal-review.googlesource.com/156883 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-02-20Add 'build' settings option and xbuddy image format.cmtice
This patch updates Crosperf to add the 'build' tag in the experiment file, and to allow that field to contain xbuddy syntax for using trybot and/or official builds in the test runs. It also adds a bit more checking to make sure we have everything necessary for running 'cros flash' before attempting to use it. BUG=None TEST=I have run this using an experiment file that compares a local image, a trybot image and an official image against each other. It all worked. Change-Id: Ia896799061508fb5718a3201b1599d8622de0b3f Reviewed-on: https://chrome-internal-review.googlesource.com/154097 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-01-30Update image_chromeos.py to use 'cros flash'cmtice
Check the chroot to see if it contains the 'cros flash' script; if so use 'cros flash' instead of 'cros_image_to_target.py' for imaging the DUT. This is the first of a series of changes to improve the imaging options in Crosperf. This script does not change any of the options or syntax for using image_chromeos.py. BUG=None TEST=Tested in chroot with & without cros_flash.py. Worked correctly in both cases. Change-Id: If8704c81aa800cab85379baa514249105fe2b83f Reviewed-on: https://chrome-internal-review.googlesource.com/153197 Reviewed-by: Han Shen <shenhan@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-01-15Add timeout to image_chromeos call to bin/cros_image_to_target.py, tocmtice
handle flaky hangs by that script. Also add 2 retries. BUG=None TEST=Ran image_chromeos.py with modification and it worked. Change-Id: Ib9f772b34862704255018bb68b563f721eb4fe3e Reviewed-on: https://chrome-internal-review.googlesource.com/151776 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2013-03-15Cleaned up directory after copy of tools from perforce directoryLuis Lozano
Got rid of stale copies of some tools like "crosperf" and moved all files under v14 directory (that came from perforce) into the top directory. BUG=None TEST=None Change-Id: I408d17a36ceb00e74db71403d2351fd466a14f8e Reviewed-on: https://gerrit-int.chromium.org/33887 Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org>
2012-12-20Synced repos to: 64740Ahmad Sharif
2012-10-09Synced repos to: 63271Ahmad Sharif
2012-05-07Synced repos to: 60208Ahmad Sharif
2012-02-02Synced repos to: 58208Ahmad Sharif