aboutsummaryrefslogtreecommitdiff
path: root/buildbot_test_toolchains.py
AgeCommit message (Collapse)Author
2016-04-14[crosperf] Fix more lint errors.Caroline Tice
BUG=chromium:567921 Test=Tested in my directory. Change-Id: Ibe128dd030887e0372380fb8f95a5e37ee63feeb Reviewed-on: https://chrome-internal-review.googlesource.com/255303 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-02-21Improve search for non-AFDO images for performace comparison.Luis Lozano
To compare O2 vs AFDO, we try to use images from the chrome-pfq. We now search the PFQ image created by the previous version of chromeos since that is the one that validated the chrome for the current version. If we dont find a previous, we try 2 versions before. BUG=None TEST=White testing.ran simplified version of buildbot_test_toolchains. Change-Id: I800255508117c7aa7f91468243e4713432e5561a Reviewed-on: https://chrome-internal-review.googlesource.com/249245 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org>
2016-01-09Check that non AFDO images exist before trying to use them.Luis Lozano
Check the non-AFDO images exist before we try to get performance numbers for them. BUG=None TEST=white box testing. Change-Id: I182c9da1dd29c377b98e3012ba7b5e186cdf2e4f Reviewed-on: https://chrome-internal-review.googlesource.com/243956 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2016-01-07toolchain-utils: check vanilla image before calling crosperf.Yunlian Jiang
In some cases, the vanilla image builds finishes later than the image from trybot, this could cause the nightly test job fail. This CL makes sure the vanilla image is ready before calling crosperf. BUG=None TEST=None Change-Id: Id2f28717c5a8c6c6db0d6c2e240c687199ea6a7c Reviewed-on: https://chrome-internal-review.googlesource.com/243618 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-28Run experiments on non-AFDO image in our nightly testing.Luis Lozano
We need to compare performance of AFDO vs non-AFDO built images. BUG=None TEST=in progress Change-Id: I27f4cca5d3852521ed160aa7ee7f7576c1781e0a Reviewed-on: https://chrome-internal-review.googlesource.com/243135 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Han Shen <shenhan@google.com>
2015-12-16Run pyformat on all the toolchain-utils files.Luis Lozano
This gets rid of a lot of lint issues. Ran by doing this: for f in *.py; do echo -n "$f " ; if [ -x $f ]; then pyformat -i --remove_trailing_comma --yapf --force_quote_type=double $f ; else pyformat -i --remove_shebang --remove_trailing_comma --yapf --force_quote_type=double $f ; fi ; done BUG=chromium:567921 TEST=Ran simple crosperf run. Change-Id: I59778835fdaa5f706d2e1765924389f9e97433d1 Reviewed-on: https://chrome-internal-review.googlesource.com/242031 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-12-14buildbot_test_toolchains.py: clean lint warning.Yunlian Jiang
This cleans up lint warnings and changes the condition to decide whether it is a clang job. BUG=chromium:567921 TEST=it is lint warning free. Change-Id: I0742b57c3f6bf6d5839e58d28907ded45d1ee40a Reviewed-on: https://chrome-internal-review.googlesource.com/241985 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-12-08Re-enable graphics_WebGLAquarium from buildbots testing.Luis Lozano
With some recent changes to the test_that invocation this should work now. Still there is an issue with benchmarks that require compilation for the particular board. That remains to be fixed. BUG=chromium:551710 TEST=None. This is a revert. Change-Id: I547a4853302f11dc58ddc493ee5368b1e2d33c0e Reviewed-on: https://chrome-internal-review.googlesource.com/241035 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-11-05Temporarily remove graphics_WebGLAquarium from buildbot testing.Luis Lozano
For the images that we build in trybots, this test cannot be executed because we are not downloading the necessary artifacts for autotest. See chromium:551710. This change temporarily removes that test from the nightly testing that we do based on trybot images. These changes should be removed once this bug is fixed. BUG=chromium:551710 TEST=None Change-Id: I15ec9386d52809492345d9cd46a6f0ad39330dbc Reviewed-on: https://chrome-internal-review.googlesource.com/237838 Commit-Ready: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-11-05Make 'label' names more meaningful in nightly tests.Caroline Tice
The nightly crosperf tests currently use the label 'test_image' for the images that we build or that the trybot built. This is not a very informative name. This CL tries to make the names a least a little bit more useful. BUG=None TEST=Tested by hand in role account directory tree. Change-Id: Ib88f711ec96ec5cb71efebae79a76e6e8ebc784a Reviewed-on: https://chrome-internal-review.googlesource.com/237796 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-09-25Add compiler info to experiment data.Caroline Tice
Check the build label to see which compiler was used. Store the compiler in the label, and add it to the json filename and output. Also update nightly buildbot script to add 'compiler' field to labels. BUG=None TEST=Tested in my directory. Change-Id: I124c4910cce1145708bb2a52e3a6d6dff93777c3 Reviewed-on: https://chrome-internal-review.googlesource.com/231811 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-09-14Make new scheduler the default.Han Shen
This CL also add a new option '--noschedv2' to use the original scheduler in case of bugs in the new scheduler. Change-Id: I11ddefe7e7c1162019e9ad72ed3cef0b962b4d06 Reviewed-on: https://chrome-internal-review.googlesource.com/230484 Commit-Ready: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-09-04Start generating json archive files for nightly tests.Caroline Tice
This turns on the option for the nightly tests to generate json files, which get stored in a pending archives directory and can later be picked up, converted to ColumnIO and stored on a colossus server. It also fixes a bug where the nightly test was looking for a .boto file in /home/mobiletc-prebuild instead of /usr/local/google/home/mobiletc-prebuild. BUG=None TEST=None Change-Id: Id413097686d8e72499d141ca7b8d2e131a993cb9 Reviewed-on: https://chrome-internal-review.googlesource.com/228860 Commit-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Luis Lozano <llozano@chromium.org>
2015-09-02buildbot_test_toolchains: differenciate llvm buildYunlian Jiang
This changes the email title and the name of experiment file when using the patch for LLVM build. BUG=none TEST=it did not die at first when running a test job. Change-Id: Iaeafbd68eaa1e30d4e5ef5edc51bfa41f5f2e56e Reviewed-on: https://chrome-internal-review.googlesource.com/228657 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-08-28Add a new option '--schedv2' so later invocation to crosperf uses the new ↵Han Shen
scheduler. This option is added to buildbot_test_toolchains.py & test_toolchains.py. TEST=None Change-Id: I8f31d0dd7b4b1db6af35759dd476f0f55e224366 Reviewed-on: https://chrome-internal-review.googlesource.com/228645 Commit-Ready: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-08-26buildbot_test_toolchains.py: small refactoring.Yunlian Jiang
This is some refactoring of this script. It only copy to weeking report if the board is lumpy, which is the only board that we report weekly result. BUG=none TEST=tested manually on mobiletc-prebuild. It does not die at begining. Change-Id: Ib270e96ff2328030f316f9388cb43db97378da62 Reviewed-on: https://chrome-internal-review.googlesource.com/227832 Commit-Ready: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-08-24buildbot_test_toolchains: fix a typoYunlian Jiang
BUG=none TEST=it runs with a patch parameter. Change-Id: I08437d0ec47ef27a3b88930358f3c149fd955cdc Reviewed-on: https://chrome-internal-review.googlesource.com/227861 Reviewed-by: Han Shen <shenhan@google.com> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com>
2015-08-21buildbot_test_toolchains: fix several errorsYunlian Jiang
This fixes some errors introduced by previous patch. BUG=none TEST=the script is running fine on chrotomation2 Change-Id: Id6b49941aef5f5c8902dae9dfeefe2acc51ef310 Reviewed-on: https://chrome-internal-review.googlesource.com/227840 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com>
2015-08-21buildbot_test_toolchains: make patch number a parameter.Yunlian Jiang
We want to use the buildbot_test_toolchains.py to measure the performance gap between gcc and clang. So we need a different patch to create the image built with clang. BUG=none TEST=buildbot_test_toolchains.py now accepts --patch Change-Id: I45dfd79630cea217bc6134802270c6e13b4121c1 Reviewed-on: https://chrome-internal-review.googlesource.com/227806 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Yunlian Jiang <yunlian@google.com> Tested-by: Yunlian Jiang <yunlian@google.com>
2015-08-20Finish removing temporary workaround.Caroline Tice
Chrome in ChromeOS chroot for nightly buildbot tests now has a good version of telemetry, so we can remove the temporary workaround for the buildbots. BUG=None TEST=Ran telemetry by hand inside chroot & crosperf outside chroot. Change-Id: I38db7d9a2f427df3cc29418bc866d236e213247b Reviewed-on: https://chrome-internal-review.googlesource.com/227608 Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Reviewed-by: Yunlian Jiang <yunlian@google.com>
2015-08-12Temporarily use good external Chrome source tree.Caroline Tice
This is to work around a problem with the current version of Chrome inside ChromeOS (telemetry is broken in that version of Chrome). This CL makes the nightly tests use an external Chrome source tree that has been verified to work properly. This will be un-done, once ChromeOS gets a good version of Chrome. BUG=None TEST=Tested crosperf on chrotomation2 using this. Change-Id: I41caef9bb1b1973c83745e3962a5c36c489d7514 Reviewed-on: https://chrome-internal-review.googlesource.com/226695 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org>
2015-06-30Update all_toolchain_perf to use new recommended set.cmtice
Update the all_toolchain_perf alias to use the new recommended set of tests. Also update the default test values to include default values for the newly recommended tests. Also update the nightly tests to use the newly recommended set. BUG=None TEST=Tested new alias with crosperf. Change-Id: I603f866723646a2d5b6d86fbe1de3c4617e09046 Reviewed-on: https://chrome-internal-review.googlesource.com/220385 Reviewed-by: Han Shen <shenhan@google.com> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com>
2015-06-23Remove temporary workaround for nightly tests locking.cmtice
Due to issues with mobiletc-prebuild not being able to access the afe locking mechanisms, the nightly tests were temporarily using the old file locking mechanism. The issues with the role account using the afe locks has been resolved, so this CL reverts the nightly tests to using the afe machine locking mechanism. BUG=None TEST=Ran crosperf tests on chrotomation2 from role account using afe locking mechanism. Change-Id: I243c28d80718c2e98913ad49d50bf5b46ff82758 Reviewed-on: https://chrome-internal-review.googlesource.com/219905 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-06-17Add automatic retries to nightly testing.Luis Lozano
Used new "retries" crosperf functionality in our nightly testing. BUG=None TEST=None (already tested in crosperf side) Change-Id: I42b9d04ef690593485df825b2a3aa41b9e2f6cc3 Reviewed-on: https://chrome-internal-review.googlesource.com/219310 Reviewed-by: Han Shen <shenhan@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org>
2015-06-12Update nightly tests to use old file locking mechanism for now.cmtice
There are some issue with the role account using the AFE server locking mechanism, so we will temporarily revert the nightly tests to use the old file locking mechanism. This CL depends on the CL that re-enables the file locking mechanism (with a new flag). BUG=None TEST=None CQ-DEPEND=CL:219096 Change-Id: If525e4fcf292d3c57aa9c69a1b796914bd6d434c Reviewed-on: https://chrome-internal-review.googlesource.com/219088 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-05-27Update nightly tests to use mail-sheriff program for reports.cmtice
This updates the nightly buildbot and local tester programs to tell crosperf to put its results in a special directory, and to not send email. After crosperf finishes, it looks for the email information in the special directory and sends out then email using the mail-sheriff program. BUG=None TEST=Currently running buildbot test. Will not commit the change until the test passes. Since the test takes several hours to run I wanted to put this out for review now. Change-Id: Idd3d137e4a6e144b677690fc6775580dfac56af1 Reviewed-on: https://chrome-internal-review.googlesource.com/217286 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-04-24Make sure latest toolchains are build in non vanilla testing.Luis Lozano
Our non-vanilla testing builds with latest compiler but it was not using --latest_toolchain argument for cbuildbot. Also, did some minor cleanup. BUG=None TEST=None (will let it test in the nightly testing) Change-Id: I7316fcd8b9548f7038b110bdf6cd4f60714f41d2 Reviewed-on: https://chrome-internal-review.googlesource.com/214363 Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@google.com>
2015-02-19Only tar chromiumos_test_image.bin file, instead of all files under images ↵Han Shen
directory. Change-Id: I27ca96a5e347de6d2e5ae280cfd37a7669056679 Reviewed-on: https://chrome-internal-review.googlesource.com/199736 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Han Shen <shenhan@google.com> Tested-by: Han Shen <shenhan@google.com>
2015-02-14Mark buildbot_test_toolchains.py, test_toolchains.py, and weekly_report.py ↵Rahul Chaudhry
executable. BUG=None TEST=None Change-Id: Ia49aef079414d7072a57dc14d9b0d99fa3a75d40 Reviewed-on: https://chrome-internal-review.googlesource.com/198670 Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Rahul Chaudhry <rahulchaudhry@google.com> Tested-by: Rahul Chaudhry <rahulchaudhry@google.com>
2015-02-14Add date to job descriptions; wait for pending builds.cmtice
Previously if a trybot job got put into a pending queue rather than starting immediately, we didn't notice and would end up using the previous day's trybot image. This CL fixes that. BUG=None TEST=Ran successfully from role account. Change-Id: I3dd2cf1edd639e562fa78769c86c99e4c51f3a0b Reviewed-on: https://chrome-internal-review.googlesource.com/198610 Reviewed-by: Yunlian Jiang <yunlian@google.com> Reviewed-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2015-02-05Make fatal errors if trybot job fails or trybot image is empty.cmtice
Update GetTrybotImage to check the final trybot job status and return an error if the job did not have a success status. Also update the nightly buildbot test jobs to fail fatally if the trybot or vanilla image names are empty. BUG=None TEST=Tested this in role account. Change-Id: I17c4ca5fa77ea488c8bf3ae4e5c225b747c90c87 Reviewed-on: https://chrome-internal-review.googlesource.com/196095 Reviewed-by: Yunlian Jiang <yunlian@google.com> Commit-Queue: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com>
2014-12-18New script for using buildbots for nightly tests.cmtice
This script could eventually replace the current test_toolchains.py script. Instead of downloading and building ChromeOS locally, it launches a buildbot to build ChromeOS with the latest mobile toolchain compiler. It then downloads the trybot image (and the corresponding vanilla image), generates the crosperf experiment file, and launches crosperf. I have been running this as a second cron job on chrotomation2, running daisy tests, and it has been working just fine. BUG=None TEST=Ran in cron jobs on chrotomation2. Change-Id: I3b17d0373e2fae359fc8f4e0188dcec2943c7b71 Reviewed-on: https://chrome-internal-review.googlesource.com/187735 Reviewed-by: Caroline Tice <cmtice@google.com> Trybot-Ready: Caroline Tice <cmtice@google.com> Tested-by: Caroline Tice <cmtice@google.com> Commit-Queue: Caroline Tice <cmtice@google.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-10 23:21:24 +0000