aboutsummaryrefslogtreecommitdiff
path: root/utils
AgeCommit message (Collapse)Author
2016-02-01toolchain-utils: remove uses of deprecated-module 'optparse'.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I0e239a91abad72920c0d5e6e70b43683ecccb868 Reviewed-on: https://chrome-internal-review.googlesource.com/246640 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-02-01toolchain-utils: fix lint warnings related to table_type argument.Rahul Chaudhry
This CL fixes docstring-missing-args and no-value-for-parameter lint warnings related to the table_type argument. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I2d3b38b53efb78d39adb78d2c51015bc117b9c31 Reviewed-on: https://chrome-internal-review.googlesource.com/246639 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-02-01toolchain-utils: fix docstring-* lint warnings.Rahul Chaudhry
This CL fixes docstring-section-newline, docstring-arg-spacing, and docstring-cuddled-quotes lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I20c94d72199920ef1f78395dcc3f2ac051fc33cd Reviewed-on: https://chrome-internal-review.googlesource.com/246638 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-28misc: Fix escaping of some regular expressionsDavid Sharp
The characters in the last pattern of GetFilenameFromString were particularly mangled, and probably do not implement what was originally intended. This change has it delete any of r"\" (backslash), "$", "=", '"' (double quote), "?", or "^". Also replace assertTrue(a == b) with assertEqual(a, b) in tests. TEST=unit test BUG=none Change-Id: I9c12ce585e5a93be90a8e4054754115c47fdb7b2 Reviewed-on: https://chrome-internal-review.googlesource.com/246158 Commit-Ready: David Sharp <dhsharp@google.com> Tested-by: David Sharp <dhsharp@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-23toolchain-utils: disable protected-access lint warnings in tests.Rahul Chaudhry
crosperf/experiment_runner_unittest.py and crosperf/machine_manager_unittest.py had a combined total of 53 protected-access lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: Icc31a63f3383068620314fd440460ab82df463a5 Reviewed-on: https://chrome-internal-review.googlesource.com/244868 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-23toolchain-utils: fix class-missing-docstring lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I0640cab2c8d15380badbfbc0db39e3063cc85471 Reviewed-on: https://chrome-internal-review.googlesource.com/244867 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-23toolchain-utils: fix module-missing-docstring lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: Ifda134bf5f26db04551b915c189814b86e750df3 Reviewed-on: https://chrome-internal-review.googlesource.com/244866 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-09Check that non AFDO images exist before trying to use them.Luis Lozano
Check the non-AFDO images exist before we try to get performance numbers for them. BUG=None TEST=white box testing. Change-Id: I182c9da1dd29c377b98e3012ba7b5e186cdf2e4f Reviewed-on: https://chrome-internal-review.googlesource.com/243956 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-01-07toolchain-utils: check vanilla image before calling crosperf.Yunlian Jiang
In some cases, the vanilla image builds finishes later than the image from trybot, this could cause the nightly test job fail. This CL makes sure the vanilla image is ready before calling crosperf. BUG=None TEST=None Change-Id: Id2f28717c5a8c6c6db0d6c2e240c687199ea6a7c Reviewed-on: https://chrome-internal-review.googlesource.com/243618 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-01-05Modify pre-push hook to run pyformat.Luis Lozano
Run pyformat before git push. Give error if files need formatting. Also, made silly cleanup to pstat.py to be able to test the pre-push hook. BUG=None TEST=white box testing Change-Id: I79f12fa0acca4048b551f7dab25b0de8d5f05d75 Reviewed-on: https://chrome-internal-review.googlesource.com/243113 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-04toolchain-utils: fix docstring-second-line-blank lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I5b8e7fafbfb73abedb03a1d01267b4216493773b Reviewed-on: https://chrome-internal-review.googlesource.com/243456 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-04toolchain-utils: fix line-too-long lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: Ic4bd934dfb0b83e879a6f0c8a1ec121ac0a7de4b Reviewed-on: https://chrome-internal-review.googlesource.com/243414 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-04toolchain-utils: fix unused-argument lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: Iae6add81a1fb1c929a09808b5870d2a1762ef291 Reviewed-on: https://chrome-internal-review.googlesource.com/243413 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-04toolchain-utils: fix missing-print-function lint warnings.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I2e0cf0d903b0ca7662132528ee6860ea8eaee83b Reviewed-on: https://chrome-internal-review.googlesource.com/243412 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2016-01-04toolchain-utils: apply pyformat to the source files.Rahul Chaudhry
BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I0cb94c77c26941fafb97cca2b9e9bb12cf4c210e Reviewed-on: https://chrome-internal-review.googlesource.com/243411 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-29Avoid lint of couple of files we did not author.Luis Lozano
colortrans.py, pstat.py and stats.py were not written by us and we are not maintaining them. They don't comply to our coding conventions. So, it does not make sense to lint them. BUG=chromium:570456 chromium:567921 TEST=None (trivial) Change-Id: I20469396a55f6d5da0d73f5c7d33204fe6728c33 Reviewed-on: https://chrome-internal-review.googlesource.com/243258 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-28toolchain-utils: remove superfluous parentheses.Rahul Chaudhry
This CL fixes all superfluous-parens lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I96cabd8e348787fbb86da221dd3d6f04ce210041 Reviewed-on: https://chrome-internal-review.googlesource.com/243097 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-28toolchain-utils: remove unused variables.Rahul Chaudhry
This CL fixes all unused-variable lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I4f38ca9a428738125b09dc38b4ad8778dab77886 Reviewed-on: https://chrome-internal-review.googlesource.com/243119 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-28toolchain-utils: fix wrong continued indentation.Rahul Chaudhry
This CL fixes all bad-continuation lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I8bb9cb6eb60f169a63c2e3e0f717924463da2a89 Reviewed-on: https://chrome-internal-review.googlesource.com/243257 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-28toolchain-utils: fix shebang related lint warnings.Rahul Chaudhry
This CL fixes all missing-shebang and bad-shebang lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I41bd3516496b246e4e4a8ccc9f97fb25007738a7 Reviewed-on: https://chrome-internal-review.googlesource.com/243136 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-28toolchain-utils: remove unused imports.Rahul Chaudhry
This CL fixes all unused-import lint warnings. BUG=chromium:570458 TEST='crosperf/run_tests.sh' passes. Change-Id: I145cebe61117388e729cf1b67001cb727fee1b33 Reviewed-on: https://chrome-internal-review.googlesource.com/243108 Commit-Ready: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-22crosperf: fix lint warnings.Yunlian Jiang
BUG=chromium:570454 TEST=unittest passes, lint warning is gone. Change-Id: I230e9de9c1912f05774444d15310ad70e07e264b Reviewed-on: https://chrome-internal-review.googlesource.com/243215 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-15Fix arguments to RunCommand from RunCommands.Caroline Tice
Under the recent RunCommand changes, all the arguments to RunCommand except the command itself must be keyword args. The initial change missed fixing this in the RunCommands function, which resulted in an assertion error in the overnight test. This CL fixes that issue. BUG=None TEST=Tested running setup_chromeos with the role account (where previous failur occurred). Change-Id: I73834309afc5cdbddd8953de5287bf298fb80b7e Reviewed-on: https://chrome-internal-review.googlesource.com/241980 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-12-15crosperf: RunCommand should return one type of object.Luis Lozano
Cleaned up the interfaces for the RunCommand routines. These were returning different types (int or tuple) depending on the value of the return_ouput parameter. Returning different unrelated types from a routine is bad practice. Linter complains about this with several warnings like this: "Attempting to unpack a non-sequence defined at line XY of utils.command_executer" BUG=chromium:566256 TEST=ran crosperf with a example experiment file Ran run_tests. Change-Id: Ibb83ab9322c87558077fc4937ef5c0686bbe5417 Reviewed-on: https://chrome-internal-review.googlesource.com/241459 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Han Shen <shenhan@google.com>
2015-10-29buildbot_utils: make it accept multiple patches.Yunlian Jiang
BUG=None TEST=it accept multiple patches Change-Id: I29154f79df4200ab5f03134e0a84624c75ad9b13 Reviewed-on: https://chrome-internal-review.googlesource.com/236752 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-10-02Fixed several warnings reported by pylint.Luis Lozano
Fixed most of warnings in command_executer.py. There are a couple that are harder to fix and will require changes in many files. Was doing this to get an idea of how difficult would it be to fix all the warnings in toolchain-utils directory. BUG=None TEST=Ran some benchmarks afterwards. Change-Id: I11eec9e86317431b8ed7779d17e2dfae1be9f594 Reviewed-on: https://chrome-internal-review.googlesource.com/232829 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: David Sharp <dhsharp@google.com>
2015-10-01Fixed some issues with signal handling and cleanup.Luis Lozano
Fixed several issues related to signal handling and cleanup: - Got rid of the extra process wrapper around crosperf. We now do an "exec" to get rid of this extra process. This facilitated signal handling. - Found a better fix for the problem we had with ignoring ctrl-c. Instead of creating a pipe for child process stdin, we now create the child with setsid which has the side effect of creating the child without a terminal associated to it (see man setsid). - By creating the child with setsid, we also allow for easier killing of the child and its descendants since it allows us to use killpg (see man killpg). - Handle SIGTERM by "converting" it into an exit call. This will make sure the atexit routines are called and, furthermore, we can terminate child process when you catch the "SystemExit" exception. - With these changes, crosperf will send a SIGTERM to any cros_sdk, test_that processes that it has started. Before, we would have left over test_that processes that could interfere with a new test_that invocation. BUG=chromium:492826 BUG=chromium:470658 TEST=verified by hand signals are working. Ran toolchain suite. Change-Id: Ibd09ff428a7b402ea2296ecf9b9a34aa5756c93f Reviewed-on: https://chrome-internal-review.googlesource.com/232616 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-09-29Fixed ignoring of ctrl-c.Luis Lozano
Found out that if I send the signal directly to the crosperf process by "kill -SIGING <pid>", crosperf handles it correctly. However, the ctrl-c from the keyboard did not work. I started tracing it and found out that crosperf shares the stdin with the children process. If I avoid this sharing of stdin everything works fine. I assume one of the commands executed is modifying some settings in the terminal and changes the keyboard mapping for SIGINT. To fix it, when invoking popen, request the creating of a pipe to be used as the stdin for the new process. Once popen returns close this pipe to avoid having the child process blocked on this pipe. BUG=chromium:492826 TEST=Ran a few tests in crosperf. Verified the ctrl-c now works. Change-Id: I767054a6f9e43a5137a98e074515b41db9a222ab Reviewed-on: https://chrome-internal-review.googlesource.com/232011 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-09-24Add arm64 as a valid architecture.Caroline Tice
BUG=None TEST=Tested in my directory (with build_tc.py --board=arm64-generic) Change-Id: Ib4d0e61a77c279c2c47c5d8e6ac6b71af4adb25e Reviewed-on: https://chrome-internal-review.googlesource.com/232166 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-08-28Fixed unbound variable problem.Luis Lozano
Variable "to_be_deletes" may not be initialized in cases where os.write before its initialization fails. Fixed by moving out initialization out of the try block. Also fixed a few pylint warnings. BUG=chromium:526186 TEST=None Change-Id: I0aaada439d9145c0f37fae877b82a03ddb4fb476 Reviewed-on: https://chrome-internal-review.googlesource.com/228503 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-08-07Deal properly with non-unique buildbot build ids.Caroline Tice
In the new buildbot waterfall arrangement, occasionally a buildbot build id will not be unique. This was causing two different problems for us, for a buildbot id that is not unique: 1. If our job finished successfully and generated a build image artifact, we end up finding multiple artifacts, and we were using the first one founs (which was the earliest). 2. If the job failed to generate a build artifact, we would still successfully find a build artifact from an earlier build with the same id. This CL fixes both problems by 1. Checking the build status and only looking for a build artifact if the job looks like it succeeded; 2. Sorting the build artifacts we find, to get and use the most recent. BUG=None TEST=Tested on the build logs with all the various cases that failed improperly today. Change-Id: I627aa48ae13d3c115acd6cb3c411e382f0a8f1c6 Reviewed-on: https://chrome-internal-review.googlesource.com/226096 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Han Shen <shenhan@google.com>
2015-08-06buildbot_utils: for release builds, get logs from the 'release' builder.Rahul Chaudhry
The individual ${BOARD}-release builders are now grouped together into one 'release' builder. BUG=None TEST=None Change-Id: Ie0a2ea1298c63f71385f4f9331e611d8ac22efca Reviewed-on: https://chrome-internal-review.googlesource.com/225253 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com>
2015-08-05buildbot_utils: fix waiting logic in GetTrybotImage.Rahul Chaudhry
The problematic case is when buildbot_json.py execution fails and the call to GetBuildInfo returns nothing (line 209). done is set to False (line 216), and the full "if done:" block is skipped. Unfortunately, the "if not done:" block starting at line 242 is inside the "if done:" block (Yay Python!). This simply jumps back to the beginning of the while loop (line 207) without sleeping for SLEEP_TIME and without updating pending_time. Result: an infinite loop with no delay spawning a new process on each iteration. BUG=None TEST=SLEEP_TIME and TIME_OUT are respected in the loop. Change-Id: I7cc741b7a183233507968277345a079aecce9cd2 Reviewed-on: https://chrome-internal-review.googlesource.com/225252 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com>
2015-06-26Only search buildbot logs for appropriate builders.cmtice
Update the buildbot scripts to only search for build records on the builders we launched the jobs on. Previously the scripts were searching the build logs of all builders. BUG=None TEST=Tested with and without this on chrotomation2. Without this it took 16 minutes; with it it took 1 minute. Change-Id: Id36e6fc5688299f3af765b439de66a029b1f896e Reviewed-on: https://chrome-internal-review.googlesource.com/220137 Reviewed-by: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Mungyung Ryu <mkryu@google.com>
2015-06-23Add debugging/triaging message to buildbot job.cmtice
Check the buildbot launch results to make sure the buildbot launched successfully before continuing. Otherwise check pending time and either sleep some or issue fatal message (if pending time times out). BUG=None TEST=Tested by hand in my directory. Change-Id: I27bd6bf4fb88f0d9db584272e74f9a2ad31f9381 Reviewed-on: https://chrome-internal-review.googlesource.com/219727 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-06-10Implement new global locking scheme for machines.cmtice
This CL implements a new machine locking mechanism using Autotest Front End servers. When locking/unlocking a lab machine, it uses the ChromeOS HW lab server; when locking/unlocking a local machine, it uses a local AFE server on chrotomation2. BUG=None TEST=Tested the script manually to lock/unlock machines and query status. Also tested with image_chromeos and with crosperf scripts. Change-Id: I2793bc1f7dc056e725694e81ded656d9f49d227b Reviewed-on: https://chrome-internal-review.googlesource.com/217560 Reviewed-by: Luis Lozano <llozano@chromium.org> Reviewed-by: David Sharp <dhsharp@google.com> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com>
2015-05-22Remove run_remote_tests references.cmtice
run_remote_tests.sh is about to be removed from ChromeOS. This CL removes all references to run_remote_tests from toolchain-utils. It also updates unittests to make sure they work with recent changes. BUG=crbug:498778 TEST=Verifed results_cache_unittest (and all the other unittests) works with changes. Change-Id: Ic9413bf77a1679e1267b81b182fa7f64da2629b5 Reviewed-on: https://chrome-internal-review.googlesource.com/216884 Reviewed-by: David Sharp <dhsharp@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-05-21Fix confusing error message when cherrypy is not installed on DUT.Luis Lozano
Also fixed a few warnings. BUG=None TEST=Installed a couple of images on machines with and without cherrypy. Change-Id: I40d3fa1882e3bceb0e70a8d040a892e894cf50db Reviewed-on: https://chrome-internal-review.googlesource.com/217045 Reviewed-by: Caroline Tice <cmtice@google.com> Tested-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org>
2015-05-16Update email sender to use sendgmr, if it's available.cmtice
BUG=None TEST=Tested with crosperf tests on my system. Change-Id: I4db4cd70d74e5f13b1fe8bb022e5403f6902d1ee Reviewed-on: https://chrome-internal-review.googlesource.com/216312 Reviewed-by: David Sharp <dhsharp@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-04-28Use "http://chromegw/i/tryserver.chromiumos/" for retrieving builder info.Han Shen
This "/i/" url is the internal version of "http://chromegw/p/tryserver.chromiumos/". And "/i/" links offer waterfall controls (force build, stop build, etc.) and work for both internal and public waterfalls, whereas "/p/" links only work for public waterfalls. BUG=None TEST=buildbot_json.py works well with this url. Change-Id: I55bebdb53020447156df2f7477dd3d9a32478660 Reviewed-on: https://chrome-internal-review.googlesource.com/214840 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com>
2015-04-26Fix current problem with machine locks.cmtice
Currently if crosperf can't access the default locks directory, it assumes that it is safe to proceed without locks and it uses machines without checking locks. This script does two things: It causes crosperf to fail if it can't access a locks directory; and it allows the user to specify an alternate locks directory. As a by-product it also adds a boolean prompting function to utils/misc.py BUG=None TEST=Tested with default directory accessible & inaccessible; tested using local locks directory. Change-Id: I6f24f772986813bf089f5f4e40335a15e8ee4837 Reviewed-on: https://chrome-internal-review.googlesource.com/214129 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-04-24Fix typo in previous commit.Luis Lozano
BUG=None TEST=None Change-Id: I08b51a5e716df35c768e64de95ae386937f14b12 Reviewed-on: https://chrome-internal-review.googlesource.com/214487 Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-04-24Make sure latest toolchains are build in non vanilla testing.Luis Lozano
Our non-vanilla testing builds with latest compiler but it was not using --latest_toolchain argument for cbuildbot. Also, did some minor cleanup. BUG=None TEST=None (will let it test in the nightly testing) Change-Id: I7316fcd8b9548f7038b110bdf6cd4f60714f41d2 Reviewed-on: https://chrome-internal-review.googlesource.com/214363 Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-04-21Wait 10 minutes after builbot launch before looking for data.cmtice
This is to fix a problem where sometimes there is a slight delay in the buildbot launching, so we end up looking for the buildbot data before it's there, which causes the script to fail. BUG=None TEST=None Change-Id: Icf0206ce4012bef437a76f3ce4192461f15fd476 Reviewed-on: https://chrome-internal-review.googlesource.com/214009 Reviewed-by: Rahul Chaudhry <rahulchaudhry@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-04-10Add a new flavor of command executer.Han Shen
Why - The usefulness of this new cmd executor is that it allows programmer to examine the execution output/err stream while the binary is being executed. What we can do with our current weapons in the command_executor repertoire is to kick-off the execution, capture all the command output/err in memory, wait for its finish and examine the output in memory. A concrete example - Let me give a concrete example for this - while I was reading one of the nightly scripts, I came across this - commands = ("{0}/utils/buildbot_json.py builds " "http://chromegw/p/tryserver.chromiumos/" .format(file_dir)) _, buildinfo, _ = ce.RunCommand(commands, return_output=True, print_to_console=False) This kicks-off a long-run command line, captures all of its output in a string, and then parse to pick out some information. There is some issues with this - a) - memory consumption - we capture all output in a python string b) - no way to control the execution, what if what we are interested in is at the very beginning or in the middle of the command output, and once we get that we are done, and we don't want it to run any longer. c) - we only have hinder sight about the execution, what if the execution spit out errors like 'unable to connect, wait for 30 minutes till timeout', which we may know of and could act earlier (for example, kill it prematurely and report back to user). A better approach So here is how the proposed cmd executor works to address the above limitations - ... ... def my_line_consumer(line, output): if output == 'output': # parsing this line if output == 'err': # ignore this line return true; // keep execution, false to kill the current execution. commands = ("....") rv = ce.RunCommand(commands, my_line_consumer) # no output string is returned. ... ... The only difference is that we pass a 'line_consumer' to the cmd executor, this consumer will be fed with output/stderr (in unit of lines), and depending on line content, it decides whether to kill the execution prematurely. Current status I have this implemented (not many lines of code, < 100) in my bisecting scripts, and I plan to port it to toolchain_utils/utils. Change-Id: I2318dda796b3dcfa6ebe604091b41f9d68525d95 Reviewed-on: https://chrome-internal-review.googlesource.com/208619 Reviewed-by: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Commit-Queue: Han Shen <shenhan@google.com>
2015-02-14Add date to job descriptions; wait for pending builds.cmtice
Previously if a trybot job got put into a pending queue rather than starting immediately, we didn't notice and would end up using the previous day's trybot image. This CL fixes that. BUG=None TEST=Ran successfully from role account. Change-Id: I3dd2cf1edd639e562fa78769c86c99e4c51f3a0b Reviewed-on: https://chrome-internal-review.googlesource.com/198610 Reviewed-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-02-11Search Google Storage for trybot image artifacts.cmtice
Sometimes trybot jobs fail in the testing phase, but the build finished and a build artifact exists. To deal with this, instead of checking the trybot job status, we just search for the build artifact in google storage. This also removes the '--chrome_rev=tot' flag from the buildbot command. Change-Id: I38c07270d774258953d9b1f1e694aba9730ac92b Reviewed-on: https://chrome-internal-review.googlesource.com/197366 Reviewed-by: Luis Lozano <llozano@chromium.org> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com>
2015-02-06Fix syntax error: Replace 'else if' with 'elif'.cmtice
Change-Id: I3be3893c69925dd9e5e38be0b885ddf18dc2edd6 Reviewed-on: https://chrome-internal-review.googlesource.com/196445 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-02-05Update command_executer.py to allow a silent execution mode,cmtice
where it does not attempt to write any logs. Update auto_delete_nightly_test_data.py to use the command_executer in silent mode (so it doesn't run out of space writing logs on the disk it is trying to clean up). Change-Id: Id1c2c5f171a2d3854d3b53516ead31050da647b4 Reviewed-on: https://chrome-internal-review.googlesource.com/195877 Reviewed-by: Han Shen <shenhan@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-10 21:52:26 +0000