aboutsummaryrefslogtreecommitdiff
path: root/buildbot_test_toolchains.py
AgeCommit message (Collapse)Author
2021-09-02users of mail-sheriff: s/sheriff/detectiveGeorge Burgess IV
Starting tonight, an alias for the mail-sheriff program will be mail-detective. Swap to that. This includes forced auto-style-fixing from yapf. BUG=b:198446837 TEST=None Change-Id: I8f7375347c17095a7d53408617f8e46078f3db54 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/3137913 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: George Burgess <gbiv@chromium.org> Tested-by: George Burgess <gbiv@chromium.org>
2020-12-10toolchain-utils: Fix hyphen/underscore issue for nightly tests.Caroline Tice
The recipe builders use hyphens, while gsutil uses underscores. We had a fix in place for this for veyron_minnie; this updates the fix for veyron_tiger. BUG=None TEST=None Change-Id: Ide32264e1cbbe29afc8ca2cd44e449b56f91a2c1 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2585646 Auto-Submit: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org>
2020-10-07toolchain-utils: create pending_archives if neededDenis Nikitin
The change fixes the following error happening after migration to a new chrotomation: cp: cannot create regular file '/usr/local/google/crostc/pending_archives/.': No such file or directory Traceback (most recent call last): ... File "./buildbot_test_toolchains.py", line 259, in _CopyJson raise RuntimeError( RuntimeError: Crosperf failed to run tests, cannot copy json report! BUG=None TEST=To be tested on chrotomation.mtv. Change-Id: I4de2bb1ec816fa6d9e98993d6da1b6bded34b1f5 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2453594 Commit-Queue: Denis Nikitin <denik@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2020-05-14toolchain_utils: Make DUT config in nightly crosperf defaultDenis Nikitin
Remove experiment arguments reducing performance noise from the buildbot_test_toolchains script and make the values default in experiment settings. Add descriptions about default values. BUG=None TEST=Tested on local DUT. Change-Id: I5fa3d5bf36a2ca37da713b42b49c1f2ad4f08c98 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2197549 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2020-05-03Replace 'veyron-minnie' with 'veyron_minnie' for vanilla image lookup.Caroline Tice
The recipe builders use hyphens in the board archives, so when we get the trybot image from the recipe builder archives, it has a hyphen rather than an underscore. We then use the trybot image name to find the corresponding vanilla image. But the vanilla images are in archives that are still using the underscore in the board name (for veyron_minnie). So we need to update the trybot name to use the underscore again before looking for the vanilla image. BUG=chromium:1077618 TEST=None Change-Id: I01057a8546e87af0157514003cd63729853a1552 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2177651 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org>
2020-05-02toolchain-utils: s/_/-/g on board names in buildbot_test_toolchains.pyBob Haarman
Some boards are spelled inconsistently, having underscores in some places and dashes in others. This was causing nightly test reporting to fail for one board. This change makes use query with consistent naming by always using dashes in the board name and. BUG=chromium:1077313 TEST=Ran script with --test=True, see new URI, gsutil ls URI gives results. Change-Id: I7fabed1c85009e3b84e3c320aff0c7186db3d001 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2174834 Reviewed-by: Tiancong Wang <tcwang@google.com> Commit-Queue: Bob Haarman <inglorion@chromium.org> Tested-by: Bob Haarman <inglorion@chromium.org>
2020-04-16toolchain-utils: provide option to migrate nightly tests to recipezhizhouy
This patch provides two new options to buildbot_test_toolchain script, which launches nightly performance tests for toolchain. Option `--recipe` introduces new location and mechanism to find latest image from recipe bucket. The recipe builder for each board monitored by toolchain team builds an image every day and store results to certain gs locaton. When this option specified, the script no longer launches tryjob to get a new image but tries to access the lastest existing image in the last two days from the gs location. Option `--test` makes it easier to test locally, it blocks all locations in chrotomation machine and do not send or store unnecessary results after tests finish. A small change to lock_machine.py fixes an issue while related to swarming when testing this change. BUG=chromium:1071271 TEST=./buildbot_test_toolchains.py --board=lulu --remote=chromeos2-row9-rack8-host7.cros --chromeos_root=$chromiumos --test=True --recipe=True Change-Id: I724a31ce174955c1f002bb7e330919f23535e105 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2151717 Reviewed-by: Tiancong Wang <tcwang@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Commit-Queue: Zhizhou Yang <zhizhouy@google.com>
2020-04-07toolchain-utils: update email sending for nightly testzhizhouy
This patch provides follow up to http://crrev.com/c/2119231. Crosperf will return non-zero when terminated or at least one test failed. When tests partially succeed, reports will still be generated, otherwise no reports. We will try to access the report no matter Crosperf's return value, send email and copy json report to archive dir; and then raise error when Crosperf returns non-zero accordingly. Also add date to the email title. BUG=chromium:1063703 TEST=Simply tested the coverage with print. Change-Id: I4b3f3667cf3a0fd9cef4906052de38432684862c Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2137941 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2020-03-24toolchain-utils: Generate more detailed logs for nightly testszhizhouy
By default the logs of nightly tests do not show details about how certain tests fail. This patch increases the logging level of nightly performance test to 'verbose' explicitly and it will help people debug errors. BUG=chromium:1063703 TEST=None Change-Id: I1f3889a9816fee1172291310964505a8f5468cb6 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2117639 Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: George Burgess <gbiv@chromium.org>
2020-02-14toolchain-utils: fix cros lint and some random errorsZhizhou Yang
We have enabled --py3 for cros lint and thus reveals some new linting errors in migrated scripts. This patch fixes them and some other random errors. BUG=chromium:1011676 TEST=Passed unittests. Change-Id: If129e01c21845e1b944a2b64e50e9fed7138c845 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2055972 Tested-by: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: George Burgess <gbiv@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
2020-02-12toolchain-utils: Partially port scripts to python 3Zhizhou Yang
This patch ports some still-in-use python scripts under root directory of toolchain-utils to python 3. BUG=chromium:1011676 TEST=Passed unittests and tested with manually launching. Change-Id: Id6066944780a7204fe4746cd271f41ac20f2274d Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2049103 Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Tested-by: Zhizhou Yang <zhizhouy@google.com> Auto-Submit: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: George Burgess <gbiv@chromium.org>
2020-02-10toolchain-utils: fix bug in moving weekday testing to nightly-testing.Luis Lozano
BUG=none TEST=none Change-Id: I8477d79d409a49219d6458779c70c0da0376de69 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2046613 Tested-by: Luis Lozano <llozano@chromium.org> Auto-Submit: Luis Lozano <llozano@chromium.org> Commit-Queue: Zhizhou Yang <zhizhouy@google.com> Reviewed-by: Zhizhou Yang <zhizhouy@google.com>
2020-02-07toolchain-utils: move weekday testing into nightly-testing dir.Luis Lozano
BUG=None TEST=None. Change-Id: I99b9c1c09bcf7d6aaddd79f38e9ea8105f192eb0 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2043954 Reviewed-by: George Burgess <gbiv@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Commit-Queue: Luis Lozano <llozano@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Auto-Submit: Luis Lozano <llozano@chromium.org>
2019-12-13toolchain-utils: Correct argument typo when launching Crosperf.Caroline Tice
Use '--intel_pstate=no_hwp', rather than '--no_hwp' (not a valid argument), for launching nightly performance Crosperf tests. BUG=None TEST=Launched crosperf manually on chrotomation2 with this and it worked. Change-Id: Ia6474e3fb99bd30c79e15465dff0b54903e2a6f3 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1965433 Reviewed-by: Tiancong Wang <tcwang@google.com> Reviewed-by: George Burgess <gbiv@chromium.org> Commit-Queue: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org>
2019-11-01Nightly tests: Fix a typo in generating experiment files.Tiancong Wang
Recent change made a typo to check board name. This patch fixes the typo to resume nightly tests. BUG=None TEST=None Change-Id: Id5d17e1b1f0c09429159e03f36c5e11a1336ec20 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1895498 Tested-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: Zhizhou Yang <zhizhouy@google.com> Commit-Queue: Tiancong Wang <tcwang@google.com>
2019-10-30Nightly tests: Do not run aquarium stories in telemetry for Arm boardsTiancong Wang
This patch adds a check to skip adding aquarium stories in rendering.desktop if the board is not supported, when creating the experiment files for nightly tests. BUG=None TEST=None Change-Id: I80d5f1ae341d8ee0e41b55c180c6c5c4986a3e33 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1890821 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com>
2019-09-28crosperf: Print top5 commands into logDenis Nikitin
Enable top in nightly tests and print top 5 commands with high CPU usage into log. BUG=chromium:966514 TEST=unittests and HW tests on eve passed. Change-Id: I0efd7f0875829c4423e998ec2fbec7436f7ae148 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1829309 Tested-by: Denis Nikitin <denik@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org>
2019-09-17toolchain_utils: Reduce CPU freq in nightly testsDenis Nikitin
The change should provide more stable performance results on ARM devices by reducing CPU frequency below 95% of max freq. Experiments show that second from highest frequencies on kevin64 and scarlet reduce the noice on octane, aquarium and loading.desktop by 20-50%. BUG=chromium:966514 TEST=95% freq tested on scarlet and kevin64. Change-Id: I1e9d0e57bb95d48856a0d0ab564756cc1c7d3fbb Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1806797 Reviewed-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-09-12toolchain-utils: Add CPU cool-down in crosperf experimentDenis Nikitin
Added "cooldown_time" and "cooldown_temp" parameters in crosperf experiment file which enable CPU cool-down between benchmark iterations. The script will wait 10 minutes or until temperature goes down to 40 Celsius degree whichever happens first. BUG=chromium:966514 TEST=tested locally on eve, kevin64, veyron-minnie, scarlet. Change-Id: Ie9e28a6e8777db99d89acc0930da48613ae41857 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1802154 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-08-30crosperf: Add support for using aquarium in Telemetry.Tiancong Wang
We want to test the story aquarium in rendering.desktop and see the results comparison with graphics_WebGLAquarium in autotest. This patch adds support to filter out the most important metric to display in the report for the two aquarium stories (with different number of fish). Also fix the logic of reading histogram results when there's None value in the list. Also add the test to toolchain nightly tests. BUG=None TEST=crosperf --board=nami with the two benchmarks Change-Id: Ia1aa6bf796bee78bf6199332a07f4b96aa084845 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1779248 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-04Fix path for crosperf experiment files.Caroline Tice
The pathname used to be relative to the location of chromeos_root, but that stopped working when we moved chromeos_root and made it a symbolic link. This CL fixes the path, using the correct absolute path prefix (no symbolic links). BUG=chromium:990427 TEST=None Change-Id: I0c2bc452934c7609fdcbda8bb6ce65762ac9ca2b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1733648 Reviewed-by: Tiancong Wang <tcwang@google.com> Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org>
2019-08-02toolchain-utils: Add no_hwp to nightly perf testDenis Nikitin
Added --no_hwp argument to crosperf to run benchmark tests with intel_pstate=no_hwp kernel command line option. The option disables Intel's HW control of CPU frequency. CPU freq drops to fixed base frequency. BUG=chromium:966514 TEST=tested on samus, eve, kefka, rammus, scarlet, nami. Cq-Depend: chromium:1726635 Change-Id: I6ec2b5b04a4101d4c8ef7c0fbc5acc43f439275f Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1732431 Reviewed-by: Denis Nikitin <denik@chromium.org> Tested-by: Denis Nikitin <denik@chromium.org>
2019-07-25toolchain-utils: remove all xrangesGeorge Burgess IV
This removes all mention of xrange from toolchain-utils (modulo ones being changed in other CLs that are in flight). It's now an apparent lint error to use xrange, and it hinders our move to python3. As commented on If90d26664c70ccb73750f17573b89933fdb048f4, xrange -> range in python2 is really only a space concern (or speed in pathological cases), so migrations of this nature are generally super straightforward. I glanced at each of these callsites, and none of them appear to be pathological, so my hope is that this should all be Just Fine :) (Also fun to note that this includes a .diff file that has python code embedded in it.) BUG=None TEST=Presubmit tests Change-Id: Ic9f3ac3a5044d7a07da8a249bc505278d98203de Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1717130 Commit-Queue: George Burgess <gbiv@chromium.org> Commit-Queue: Luis Lozano <llozano@chromium.org> Legacy-Commit-Queue: Commit Bot <commit-bot@chromium.org> Reviewed-by: George Burgess <gbiv@chromium.org> Tested-by: George Burgess <gbiv@chromium.org>
2019-06-17Revert "toolchain-utils: Temporarily uses chrome head"Jian Cai
This reverts commit 441c6bcce48127e1ae717ca49af95040e9df32f7. Reason for revert: <INSERT REASONING HERE> Original change's description: > toolchain-utils: Temporarily uses chrome head > > This is to test pattern initialization on nightly and rotation builders. Will be reverted later. > > BUG=None > > TEST=None > > Change-Id: Id197e36cb8537d9d58f170f6b1b1ccb9cf836dc7 > Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1644924 > Reviewed-by: Tiancong Wang <tcwang@google.com> > Commit-Queue: Jian Cai <jiancai@google.com> > Tested-by: Jian Cai <jiancai@google.com> Bug: None Change-Id: I7df352ac80f228e823debbca108a4743564abdde Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1662757 Reviewed-by: Jian Cai <jiancai@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Tested-by: Jian Cai <jiancai@google.com>
2019-06-05toolchain-utils: Temporarily uses chrome headJian Cai
This is to test pattern initialization on nightly and rotation builders. Will be reverted later. BUG=None TEST=None Change-Id: Id197e36cb8537d9d58f170f6b1b1ccb9cf836dc7 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1644924 Reviewed-by: Tiancong Wang <tcwang@google.com> Commit-Queue: Jian Cai <jiancai@google.com> Tested-by: Jian Cai <jiancai@google.com>
2019-03-07toolchain: Increase number of iterations in toolchain nightly tests.Tiancong Wang
This patch explicitly set the number of iterations in toolchain nightly tests, instead of using default values. We are experimenting to see if this helps our nightly tests. This patch should only affects Chrome OS toolchain team. BUG=chromium:939432 TEST=Will be tested in toolchain nightly tests. Change-Id: Ic1f5d00b39324933ecc35c4adf071fc7724a9523 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/1508838 Commit-Queue: Tiancong Wang <tcwang@google.com> Tested-by: Tiancong Wang <tcwang@google.com> Trybot-Ready: Tiancong Wang <tcwang@google.com> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-12-20toolchain-utils: Add loading.desktop@@typical to nightly tests.Caroline Tice
page_cycler_v2.typical_25 is deprecated (and flaky!). The recommended replacement is loading.desktop@@typical. This CL updates our nightly performance test infrastructure to handle loading.desktop@@typical. BUG=chromium:916340 TEST=Tested with nightly run on chrotomation2. Change-Id: I6366b3a8e6a2926ad759a209cfded890e3109810 Reviewed-on: https://chromium-review.googlesource.com/1382826 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-08-21Revert "USE local lock for buildbot_test_toolchain."Ting-Yuan Huang
This reverts commit 4260aae2fe0abb937543c6cafc44f68e99de7410. Reason for revert: afe_lock_machine is fixed. Original change's description: > USE local lock for buildbot_test_toolchain. > > Currently the AFE lock machanism for our nightly test is broken, > we use local lock as a workaround for now. > > BUG=chromium:762589 > TEST=Generate the crosperf command in another simple python file > with the same code. Run the generated command line on > crotomation2 and it goes though the locking machine stage. > > Change-Id: Icd3132bc383b63aab6d6f5237a04348f11d8726d > Reviewed-on: https://chromium-review.googlesource.com/653213 > Commit-Ready: Yunlian Jiang <yunlian@chromium.org> > Tested-by: Yunlian Jiang <yunlian@chromium.org> > Reviewed-by: Caroline Tice <cmtice@chromium.org> Bug: chromium:762589 Change-Id: Ib9f787ff48953d384dd36c72811ebd0f20dd25db Reviewed-on: https://chromium-review.googlesource.com/1178897 Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Commit-Queue: Ting-Yuan Huang <laszio@chromium.org>
2018-06-07Use internal chrome source tree for telemetry.Yunlian Jiang
Now sync_chrome works for mobiletc-prebuild, there is no need to have this workaround. BUG=none TEST=ebuild-caroline $(equery-caroline w chromeos-chrome) prepare works on mobiletc-prebuild account Change-Id: I36ca9967b8b8fe5f79ca3b4dedc69271484d633f Reviewed-on: https://chromium-review.googlesource.com/1089114 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-05-25toolchain_utils: Use external chrome source tree for telemetry.Caroline Tice
When generating the nightly performance tests, specify an external Chrome source tree, since Chrome sync'ing is broken inside Chrome OS. Also remove unnecessary code that references a file that no longer exists. BUG=chromium:846508 TEST=Successfully ran nightly tests with these changes on Chrotomation2. Change-Id: I1cba94f82ad6ffb39c941a5dc3cb8f95cb833d20 Reviewed-on: https://chromium-review.googlesource.com/1072870 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-04-18[toolchain-utils] Remove one last description reference.Caroline Tice
Previous CL, removing '--remote-description' flag, missed two uses which this CL deletes. BUG=chromium:833542 TEST=In progress. Change-Id: I37d06e12af815e063d2cd0bac71737356cdd15a5 Reviewed-on: https://chromium-review.googlesource.com/1017909 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-04-16[toolchain-utils] Remove --remote-description flag.Caroline Tice
The cros tryjob command no longer recognizes this flag so we need to remove it from our scripts. BUG=chromium:833542 TEST=Tested with launching the rotating builders on chrotomation2. Change-Id: Ifcf62ae7073d84a9f58e12c29dcce5cdf9108e9f Reviewed-on: https://chromium-review.googlesource.com/1014369 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2018-03-26buildbot_test_toolchains: update pattern to match artifact urlTing-Yuan Huang
The 'trybot-' prefix is removed in the artifact URL. See bug for more details. BUG=chromium:825997 TEST=Tested locally; The re pattern accepts the new URL. Change-Id: I46667c29c60aa6bde9710bbb0f699f4939a1a451 Reviewed-on: https://chromium-review.googlesource.com/981398 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@chromium.org>
2018-03-09cros_utils/buildbot_utils: leverage `cros buildresult`Ting-Yuan Huang
Originally it sends a http request to some magic URL and parse the contents. Now there is a `cros buildresult` available for use. BUG=chromium:807029 TEST=cros_utils/buildbot_utils_unittest.py buildbot_test_toolchains.py Change-Id: I33441859afd675c1a6b70d3d1f4e2348d54cbd77 Reviewed-on: https://chromium-review.googlesource.com/954052 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-11-22filter out -tryjob when finding vanilla image.Yunlian Jiang
We cannot find vanilla image because the build name of trybot image was falco-release-tryjob, it then tries to find vanilla image of falco-release-tryjob, but it could not find it. BUG=none. TEST=vanilla image was found. Change-Id: I9024ea27f480db70ae296012a5958d8e243f3c38 Reviewed-on: https://chromium-review.googlesource.com/786315 Commit-Ready: Yunlian Jiang <yunlian@chromium.org> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@chromium.org>
2017-11-21Add '-tryjob' suffix to build name.Yunlian Jiang
We got some errors like These configs are not tryjob safe: lumpy-release Consider these configs instead: lumpy-release-tryjob when lauching tryjobs to get the image. This CL fixes that. BUG=none TEST=the build name is set to lumpy-release-tryjob. Change-Id: I30086001e3937c956983f7c31b038b4a09b3ecec Reviewed-on: https://chromium-review.googlesource.com/781441 Commit-Ready: Yunlian Jiang <yunlian@chromium.org> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-11-07[toolchain-utils] Fix security issue.Caroline Tice
Remove the chromeos-toolchain-credentials.json file, which should not be in a publicly visible repo. Update the buildbot_test_* scripts to take an argument for the directory containing the credentials file. Update the buildbot_utils.py file to use the credentials file argument, and to default to the location in our role account. BUG=782452 TEST=With some code commented out, did basic testing of flag/options to make sure they work as expected. Change-Id: I47a98a69b9ba8d2704d25abc37fe4447fa6ba343 Reviewed-on: https://chromium-review.googlesource.com/757901 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-09-29buildbot_test_toolchain: return 2 if trybot fails.Yunlian Jiang
We will print out the trybot url in the sheriff email if the trybot fails. To to that, we need to set return value to 2 if the trybot fails. BUG=chromium:767524 TEST=None Change-Id: Id3a8eb2cbcb8f85ada28ea11d4faf9113c9369ba Reviewed-on: https://chromium-review.googlesource.com/692620 Commit-Ready: Yunlian Jiang <yunlian@chromium.org> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-09-06USE local lock for buildbot_test_toolchain.Yunlian Jiang
Currently the AFE lock machanism for our nightly test is broken, we use local lock as a workaround for now. BUG=chromium:762589 TEST=Generate the crosperf command in another simple python file with the same code. Run the generated command line on crotomation2 and it goes though the locking machine stage. Change-Id: Icd3132bc383b63aab6d6f5237a04348f11d8726d Reviewed-on: https://chromium-review.googlesource.com/653213 Commit-Ready: Yunlian Jiang <yunlian@chromium.org> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-08-23Force all toolchain tests as run_local: False.Manoj Gupta
wprgo package is missing for arm32. So we cannot run these tests locally till wprgo for arm32 is provided. BUG=chromium:754374 TEST=crosperf job is created with run_local:False Change-Id: I6503017f9350684988305d8a7d6432574dab04d6 Reviewed-on: https://chromium-review.googlesource.com/629716 Commit-Ready: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Ting-Yuan Huang <laszio@chromium.org>
2017-08-18Move cbuildbot flags to tryjob flagsTing-Yuan Huang
They are now available and needn't to be pass through. BUG=chromium:756100 TEST=buildbot_test_toolchains.py Change-Id: Icece6e74faa8966067346a3a5fc111c9fdb5e431 Reviewed-on: https://chromium-review.googlesource.com/622309 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org>
2017-08-16Switching from cbuildbot to cros tryjobTing-Yuan Huang
BUG=chromium:756100 TEST=buildbot_test_toolchains.py, buildbot_test_llvm.py Change-Id: Ie275b960afdb2ed7183a763302af70c3f2bc1092 Reviewed-on: https://chromium-review.googlesource.com/617404 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-05-24Start testing compiler-rt with llvm-next.Manoj Gupta
We want to test compiler-rt performance and stability. So start testing llvm-next with compiler-rt as default using CL 513590. BUG=chromium:705071 TEST=buildbot runs with CL 513590 are fine for daisy, squawks, elm and peppy. Change-Id: Ic43ef238c7fce0cb88caabab7e6a32b1940cb860 Reviewed-on: https://chromium-review.googlesource.com/514302 Commit-Queue: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Trybot-Ready: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-04-25buildbot_test_toolchains: get latest instead green vanilla buildTing-Yuan Huang
So as to work around the problem that some builders' latest green builds are too old. BUG=chromium:714890 TEST=Tested _GetVanillaImageName locally. Change-Id: I06571f3d55743fb56977d7ef56a51fcddfdda62e Reviewed-on: https://chromium-review.googlesource.com/486086 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Trybot-Ready: Ting-Yuan Huang <laszio@chromium.org>
2017-03-20set the default compiler to llvm.Yunlian Jiang
Since we finished the migration to llvm. So we need to set the default compiler string to llvm. BUG=None TEST=The default compiler is set to llvm. Change-Id: I2d58c2850af5cc4d5f3a13e7bd2a68cd41d3ffb0 Reviewed-on: https://chromium-review.googlesource.com/457263 Commit-Ready: Yunlian Jiang <yunlian@chromium.org> Tested-by: Yunlian Jiang <yunlian@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
2017-03-08Update toolchain buildbot test scripts for llvm profile removal.Manoj Gupta
llvm profile is no longer required. Update buildbot test scripts accordingly. BUG=chromium:699594 TEST=Verified by launching dummy jobs on chrotomation2. Change-Id: Ife411414aedd427268b5c29310fa2b15457feb52 Reviewed-on: https://chromium-review.googlesource.com/451719 Commit-Ready: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@chromium.org>
2017-02-21crosperf: set recommended iterations for benchmarksTing-Yuan Huang
This CL associates estimated standard deviations to each benchmark, according to experiments. The recommended iterations can be specified by setting iterations = 0 in the experiment files. Setting it to numbers greater than 0 will override the default iterations and behaves exactly the same as before. With this change, benchmarks in all_toolchain_perf get no more than 2% margin of error within 90% of time. See crbug.com/673558 for how the standard deviations are estimated. BUG=chromium:673558 TEST=all_toolchain_perf + page_cycler_v2.typical_25 finishes in 3.5 hours for an image on chell. Change-Id: Ie2ed07878c1237ad31a8568ae3fd3fb96cd11f3b Reviewed-on: https://chromium-review.googlesource.com/424915 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org>
2017-01-31Fix WebGLAquarium fails from trybot images.Manoj Gupta
WebGLAquarium fails since it can't find autotest packages. But we don't want to download more files to chrotomation2. So reuse autotest files from vanilla image for trybot images. BUG=chromium:687262 TEST=autotest_path added to generated experiment files. Change-Id: I99b139451c2d39da928554747f714a6e536c7b40 Reviewed-on: https://chromium-review.googlesource.com/435438 Commit-Ready: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Ting-Yuan Huang <laszio@chromium.org>
2016-12-28Update ChromeOS Toolchain buildbot for llvm-next jobs.Manoj Gupta
Update toolchain buildbot script to report data for llvm-next builds. BUG=chromium:677317 TEST=None Change-Id: I45b25a735ee8ba956d38c1e45d0db2d8acd06e70 Reviewed-on: https://chromium-review.googlesource.com/424317 Commit-Queue: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Reviewed-by: Yunlian Jiang <yunlian@chromium.org>
2016-12-15buildbot_test_toolchains: Use LATEST-master as control groupTing-Yuan Huang
Rather than finding the official image with exact the same version as the trybot image, let's just use the latest one since trybot always get lastest or latest + 1. This also solves the problem that the official image specified may not exist. BUG=chromium:674652 TEST=Tested ToolchainComparator._GetVanillaImageName locally. Change-Id: I44fbceb99d8f6cf3187366c075c2a9522869b5a5 Reviewed-on: https://chrome-internal-review.googlesource.com/312516 Commit-Ready: Ting-Yuan Huang <laszio@google.com> Tested-by: Ting-Yuan Huang <laszio@google.com> Reviewed-by: Caroline Tice <cmtice@google.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-22 22:44:40 +0000