summaryrefslogtreecommitdiff
path: root/cbuildbot/stages
AgeCommit message (Collapse)Author
2015-06-05cbuildbot: Add flag to specify bootsrap directory.Prathmesh Prabhu
cbuildbot by default bootstraps in a directory under /tmp. This CL adds a flag that lets the user specify the top level directory under which all bootstrapping directories should be created. This is especially useful on mobbuild, where /tmp is mounted noexec. This will also be used on our builders where we don't want to execute code from /tmp. The default behaviour remains unchanged, so that's what developers get. BUG=chromium:490935 TEST=`cbuildbot --test-bootstrap --bootstrap-dir path/to/custom/dir \ --local -p chromiumos/chromite binhost-pre-cq` and verify that all bootstrap re-executions use the custom directory. Change-Id: Iba4bcf29bc092815b30c9ccbb5a5cfa29ef1d583 Reviewed-on: https://chromium-review.googlesource.com/274913 Reviewed-by: Paul Hobbs <phobbs@google.com> Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
2015-06-05Rename SDK overlay directory and constant.Gilad Arnold
Since we are moving away from "board-specific" overlays, this renames the relevant constants and locations so they are more generally applicable. BUG=brillo:918 TEST=Unit tests Change-Id: I960b5dde91a70921f73e1ecfda03c86f6b68cd3a Reviewed-on: https://chromium-review.googlesource.com/275004 Reviewed-by: David James <davidjames@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org>
2015-06-05sdk_stages_unittest: Remove unused variable.Gilad Arnold
Introduced by copy-paste. BUG=brillo:918 TEST=Unit tests still work Change-Id: Id7f43c5aae84b6c096aa065c50ff573ca5f1be89 Reviewed-on: https://chromium-review.googlesource.com/275003 Reviewed-by: David James <davidjames@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org>
2015-06-04Revert "image_test: Fail the build if image tests fail"Aviv Keshet
BUG=chromium:416022 TEST=None This reverts commit 6397d6409001da437be2a61e1edb16466b000b27. Change-Id: I2ed4399dfcb02f75dbafb7266fe41337be2955bd Reviewed-on: https://chromium-review.googlesource.com/275341 Reviewed-by: David James <davidjames@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org>
2015-06-04image_test: Fail the build if image tests failNam T. Nguyen
We have ignored image tests so far because some tests are difficult to really fix, e.g. the missing symbols imported by libthread_db. But recently we started whitelisting binaries such as this, and fixing other tests, image tests should be stable enough to flip the bit now. This CL de-marks ImageTestStage as a ForgivingBuilderStage. BUG=chromium:416022 TEST=trybots on {x86,arm,amd64,mipsel-o32}-generic-full Change-Id: I5a1460714413b3fa9c5a3893d41b56ba333d2e90 Reviewed-on: https://chromium-review.googlesource.com/275064 Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Nam Nguyen <namnguyen@chromium.org> Commit-Queue: Nam Nguyen <namnguyen@chromium.org>
2015-06-04cbuildbot: add option to create GCE tarballsAmey Deshpande
Adds an optional stage to cbuildbot for generating and uploading GCE-ready image tarballs. These uploads are easy to convert GCE images and then run as GCE VM instanes. In most cases this will be of limited utility, since ChromeOS images do not come with SSH running by default and GCE does not provide serial acccess. The test images, however, do run SSH on startup and can be accessed while running on GCE. The option is disabled by default. For example: $ gcloud compute images create foo-image \ --source-uri gs://${GCS_PATH}/chromiumos_test_image_gce.tar.gz $ gcloud compute instances create foo-instance --image foo-image $ gcloud compute ssh root@foo-instance [ --ssh_key_file=... ] BUG=brillo:782 TEST=None Change-Id: I4b769d3ad757dd7a8610ba6e5ce342df6e137a22 Reviewed-on: https://chromium-review.googlesource.com/264772 Commit-Queue: Amey Deshpande <ameyd@google.com> Tested-by: Amey Deshpande <ameyd@google.com> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
2015-06-04report_stages: surface a link from slave build to its masterAviv Keshet
BUG=chromium:496752 TEST=Unit tests enhanced to use fake_cidb, and use a master_build_id. Verified that unit tests are exercising new code path. Change-Id: I89497e449e29cc2c52474e37688891a086e29c48 Reviewed-on: https://chromium-review.googlesource.com/275314 Trybot-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: David James <davidjames@chromium.org> Commit-Queue: Aviv Keshet <akeshet@chromium.org>
2015-06-04Factor out image type and file name constantsAmey Deshpande
This patch adds constants for for dev, recovery and test image file names. It also creates dicts for mapping image types to image file names and vice versa. Finally, it replaces references to these strings under chromite/cbuildbot/ and chromite/lib. There are more such references under chromite/cli/, chromite/lib/ and chromite/scripts/, which will be fixed in a later CL. BUG=brillo:782 TEST=Unittests and 'lumpy-paladin' trybot passed. Change-Id: I37fb2098924a4f1196b375bea6f44b10794bbc07 Reviewed-on: https://chromium-review.googlesource.com/274920 Reviewed-by: Mike Frysinger <vapier@chromium.org> Commit-Queue: Amey Deshpande <ameyd@google.com> Tested-by: Amey Deshpande <ameyd@google.com>
2015-06-04generate_chromeos_config -> chromeos_config.Don Garrett
I originally named the module generate_chromeos_config because I intended to only use it for generate the chromeos specific configuration Json file. However, we are using it for some Chrome OS specific scripts, and won't be able to generalize them for a long time. So... rename the module to reflect how it's being used. BUG=None TEST=cbuildbot/run_tests Change-Id: I8b111c06c48181a43e01c6d494a8a493a0371718 Reviewed-on: https://chromium-review.googlesource.com/275026 Reviewed-by: David James <davidjames@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> Trybot-Ready: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org>
2015-06-03cbuildbot_run: Make site_config available to all build stages.Don Garrett
Nobody uses this new value yet, but this CL makes the site config available to all build stages. This is prep for getting rid of the cbuildbot_config global access. Follow on changes will remove references in various stages by using the new shared site_config value instead. BUG=chromium:483566 TEST=run_tests + lint Change-Id: I54532bdc5456cdbcc7eefa99e248a646bc5a23a9 Reviewed-on: https://chromium-review.googlesource.com/274871 Trybot-Ready: Don Garrett <dgarrett@chromium.org> Reviewed-by: David James <davidjames@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org>
2015-06-03image_test: Split into the lib, and the testsNam T. Nguyen
image_test module imports some third party modules. In order for us to not depend on them at bootstrap time, this CL reorganizes those imports into an image_test_lib module, and keep only the tests in image_test module. BUG=brillo:1158 TEST=unittests TEST=trybots on release groups Change-Id: Ia60e74414abf49e8f9f81fbabd28b19d59a0cadf Reviewed-on: https://chromium-review.googlesource.com/274447 Commit-Queue: Nam Nguyen <namnguyen@chromium.org> Tested-by: Nam Nguyen <namnguyen@chromium.org> Trybot-Ready: Nam Nguyen <namnguyen@chromium.org> Reviewed-by: David James <davidjames@chromium.org>
2015-06-02config_lib: Add the ability to use templates for configs.David James
This CL adds the ability to use templates for configs, but doesn't turn it on yet. It also adds tests for configs to ensure the names of the configs consistently match the template names, and adjusts configs to match. A list of changes: - Remove some incorrectly configured "*-chrome-pfq-informational" bots. These incorrectly configured bots were unused and building Chromium, not Chrome, as the name implies. These were deprecated long ago in favor of correctly configured bots. - Renamed *-telem-chrome-pfq-informational bots to telem-chromium-pfq-informational, since they build Chromium, not Chrome. - Removed pre-cq-group. As a side-effect, this converts storm-pre-cq back to a regular pre-cq rather than being a compile-only-pre-cq (the previous code had accidentally converted storm-pre-cq to be compile-only.) - Update amd64-generic-asan-paladin to actually be a paladin builder. - Removed the unused 'binary' config type. - Clarified documentation link for 'asan' builder. This doesn't clean up every inheritance bug / messy situation in the file but it cleans it up enough that it now passes the new tests. Since we haven't turned templates on yet, you can see the changes made by this CL by just looking at the config dump. BUG=chromium:494824 TEST=New unit tests. Inspection of config dump. Change-Id: I7d94923e27ec219d8f05bacfdd976e6531e3c921 Reviewed-on: https://chromium-review.googlesource.com/274236 Reviewed-by: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2015-05-30sync_stages: Broken unittest.Don Garrett
This test has been broken for a while, but we never noticed because the builders don't run with '--network'. BUG=chromium:493981 TEST=Ran test with --network. Change-Id: Id247e69b65804202871ad426527786a922459f1d Reviewed-on: https://chromium-review.googlesource.com/274119 Trybot-Ready: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Richard Barnette <jrbarnette@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org>
2015-05-29config_lib: Rename Config -> SiteConfig.Don Garrett
Config is an overloaded term, rename the Config class to something more specific. BUG=None TEST=lint + run_tests Change-Id: I5dbb40596fa47ad4ed50deb8eb86c22232968536 Reviewed-on: https://chromium-review.googlesource.com/274122 Trybot-Ready: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Shuqian Zhao <shuqianz@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org>
2015-05-29config_lib: Remove simple cbuildbot_config references.Don Garrett
This CL gets rid of all cbuildbot_config references, other than those related to GetConfig, and OverrideConfigForTrybot. It also removes most symbols from cbuildbot_config to prove they aren't being used. BUG=chromium:483566 TEST=cbuildbot/run_tests Change-Id: Id289d1aad190d865cd4c609c5550a64f9dfdd463 Reviewed-on: https://chromium-review.googlesource.com/273913 Trybot-Ready: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org>
2015-05-29SDKPackageBoardToolchainsStage: Create tarballs in dedicated directory.Gilad Arnold
This places board toolchain overlays in /tmp/sdk-board-overlays inside the chroot, instead of dumping them in the build root. This way they're easier to harvest during prebuilt upload. BUG=brillo:918 TEST=Unit tests Change-Id: I44a1b2f7fe3a9cf6f2c46153c3d1c03fd6dac8bb Reviewed-on: https://chromium-review.googlesource.com/273601 Reviewed-by: Gilad Arnold <garnold@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org>
2015-05-29SDKPackageBoardToolchainsStage: More resilient overlayfs unmounting.Gilad Arnold
Turns out that unmounting is not easy to get right, because the mount point might appear busy even if unmount was successful, and so trying to delete the directory shortly thereafter fails. One example is the use of MountOverlayContext context inside a TempDir context in sdk_stages.SDKPackageBoardToolchainsStage: the latter fails to delete the directory even thought the former "successfully" unmounted it. This fails sdk_stages_unittest.py intermittently. To mitigate this, we let MountOverlayContext do the cleanup itself via osutils.UmountDir, which has built-in retry logic exactly for this purpose. Ugly, but works. BUG=chromium:493269 TEST=cbuildbot/run_tests works Change-Id: I394d2e3d65062a05381584ccb6606ac42ff63881 Reviewed-on: https://chromium-review.googlesource.com/273922 Reviewed-by: Gilad Arnold <garnold@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org>
2015-05-28config_lib: Move GetSlavesForMaster.Don Garrett
Move this function from generate_chromeos_config to config_lib.Config, and update all call sites. The unittests related to this method seem to really be testing the configuration information, not the method itself, so they remain in generate_chromeos_config_unittest. BUG=chromium:483566 TEST=cros lint + run_tests Change-Id: Ia9b38b4c868c6800df77c819dcb7ba63448f7893 Reviewed-on: https://chromium-review.googlesource.com/272831 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org>
2015-05-27stats: add a tick_count to pre-cq-launcherAviv Keshet
This may help diagnosing issues with the pre-cq-launcher, as well as help tracing down why some counter stats seem to be getting lost. BUG=None TEST=None Change-Id: Id8c199fc824e7e10f3dc6eee2e220a864cb2471b Reviewed-on: https://chromium-review.googlesource.com/273546 Tested-by: Aviv Keshet <akeshet@chromium.org> Trybot-Ready: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Gabe Black <gabeblack@chromium.org> Commit-Queue: Aviv Keshet <akeshet@chromium.org>
2015-05-27Implement sdk_stages.SDKPackageBoardToolchainsStage.Gilad Arnold
This new SDK stage produces per-board toolchain overlay tarballs. It does so by mounting an overlayfs layer on top of the vanilla amd64-host target created in a previous stage, then invoking cros_setup_toolchains to install the necessary toolchains for each board and packaging them in a tarball. Currently this isn't yet wired and so has no effect on the builders. CQ-DEPEND=CL:272933 BUG=brillo:918 TEST=Unit tests Change-Id: I8424b3dbbb2e1db4fb30d6a97c83640709caf5fe Reviewed-on: https://chromium-review.googlesource.com/272922 Reviewed-by: Gilad Arnold <garnold@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org>
2015-05-26stats: change a few stat names and add a diagnostic oneAviv Keshet
This CL makes a cosmetic change to the build_started stat, so that the chromite stats namespace is not filled with entries for every config ever launched. It renames the pre-cq launch_count stat slightly, and adds an additional len(plan) * len(configs) counter which should exactly correspond with the number of trybot_launching actions in cidb. This is to help track down an anomaly in the launch counts. (the delta= argument is also removed from counters; statsd documentation indicates this is not necessary, and I wonder if it is responsible for the anomaly). BUG=None TEST=pre-cq-launcher tryjob Change-Id: Ie6cff9708861f25269678fba1be8d4539a54d5ae Reviewed-on: https://chromium-review.googlesource.com/273274 Trybot-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Gabe Black <gabeblack@chromium.org> Commit-Queue: Aviv Keshet <akeshet@chromium.org>
2015-05-22pre-cq: separately log speculative and mergeable statsAviv Keshet
BUG=chromium:469860 TEST=unit tests Change-Id: I50b7eaedeecababbb60763cfd4c0348e706585db Reviewed-on: https://chromium-review.googlesource.com/272743 Trybot-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Gabe Black <gabeblack@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
2015-05-22Refactor sdk_stages.SDKPackageStage for reusability.Gilad Arnold
We need to be able to reuse some of the tarball packaging logic in a new (upcoming) SDK builder stage, so refactoring for this purpose. 1) Moved some constants to the global context, replacing a string literal with a constant (SDK_TARBALL_NAME) 2) Moved tarball creation to a global function. 3) Small touch-ups. BUG=brillo:918 TEST=sdk_stages_unittest Change-Id: I95db70330e6d90ca78465f132c710e686f4fd852 Reviewed-on: https://chromium-review.googlesource.com/272749 Trybot-Ready: Gilad Arnold <garnold@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org>
2015-05-21pre-cq: upgrade a logging level to something visibleAviv Keshet
BUG=None TEST=None Change-Id: If6e8ea9ecebb49b26d34afed992e535696dfb5eb Reviewed-on: https://chromium-review.googlesource.com/272685 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org>
2015-05-21pre-cq: add debug logging around stats being thrownAviv Keshet
Some of the CL statuses are showing up, but not the full expected set. Adding some logging to help diagnose. BUG=chromium:469860 TEST=None Change-Id: Ifc5bc370751e25128765eeb1b5909de7c84e074a Reviewed-on: https://chromium-review.googlesource.com/272697 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org>
2015-05-21Revert "image tests: Fail the build if image tests fail"Mike Frysinger
This reverts commit 18c8e8fd7eb7cbebd480e053b358bb23be58a9a8. Breaks on mipsel-o32-generic due to missing symbols in its mesa drivers. This also isn't enabled in the CQ which means we're going to be letting through CLs that break on canaries and full bots. This needs to be turned on in the CQ first. BUG=chromium:490546 Change-Id: I73f331ddc98fb5fc599cbee1bde702ee8a2383b8 Reviewed-on: https://chromium-review.googlesource.com/272365 Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2015-05-21image tests: Fail the build if image tests failNam T. Nguyen
We have ignored image tests so far because some tests are difficult to really fix, e.g. the missing symbols imported by libthread_db. But recently we started whitelisting binaries such as this, and fixing other tests, image tests should be stable enough to flip the bit now. This CL de-marks ImageTestStage as a ForgivingBuilderStage. BUG=chromium:416022 TEST=trybots on release groups, image tests should be green Change-Id: I1548ed0e9451c0fe7bdd8151ac0f7840dc9799b1 Reviewed-on: https://chromium-review.googlesource.com/272270 Reviewed-by: Mike Frysinger <vapier@chromium.org> Commit-Queue: Nam Nguyen <namnguyen@chromium.org> Trybot-Ready: Nam Nguyen <namnguyen@chromium.org> Tested-by: Nam Nguyen <namnguyen@chromium.org>
2015-05-21binhost_test: Don't check incremental compatibility on PFQ.Bertrand SIMONNET
This will disable the incremental compatibility check on the PFQ only. We don't need to preserve incremental compatibility there as we will provide new, compatible prebuilts to use. BUG=chromium:489741 TEST=trybot run on binhost-pre-cq: all 4 tests are run. TEST=trybot run on master-chromium-pfq: all 3 non-incremental tests run. Change-Id: I566f55acdd695d91bc71324b1e23614682a19e57 Reviewed-on: https://chromium-review.googlesource.com/272199 Reviewed-by: David James <davidjames@chromium.org> Tested-by: Bertrand Simonnet <bsimonnet@chromium.org> Commit-Queue: Bertrand Simonnet <bsimonnet@chromium.org>
2015-05-21Remove uses of logging.warn.Bertrand SIMONNET
logging.warn is deprecated and we should always use logging.warning (http://bugs.python.org/issue13235). BUG=None TEST=Unittests. Change-Id: I8ca32a6f0d2d96baecb4a1e3f70d2273d1894a06 Reviewed-on: https://chromium-review.googlesource.com/272445 Reviewed-by: Christopher Wiley <wiley@chromium.org> Trybot-Ready: Bertrand Simonnet <bsimonnet@chromium.org> Tested-by: Bertrand Simonnet <bsimonnet@chromium.org> Reviewed-by: Steve Fung <stevefung@chromium.org> Commit-Queue: Bertrand Simonnet <bsimonnet@chromium.org>
2015-05-20Revert "cbuildbot: Don't bootstrap chromite in /tmp"Don Garrett
This reverts commit ee6b9a60a10d6c578f72e0eb8f5027d364373f85. BUG=chromium:489870 TEST=None Change-Id: I15ec483865280a9c1ef7cda93a0170aa96afbbcb Reviewed-on: https://chromium-review.googlesource.com/272284 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2015-05-19cbuildbot_config: Backport release updates to master.Dan Jacques
Port of non-release-specific parts of: https://chromium-review.googlesource.com/#/c/264929 https://chromium-review.googlesource.com/#/c/266093 - The 'internal_pfq_branch'-derived configs mark themselves as present on the release waterfall. - Allow explicitly-specified waterfall presense (i.e., don't override with default). - Fix bug with non-canary masters in 'sync_stages' (from release branch). - Fixed some formatting errors. BUG=chromium:456203 TEST=`./cbuildbot/run_tests` Change-Id: I1bbd849b7b3875601f86e08a714057112c8d002f Reviewed-on: https://chromium-review.googlesource.com/265727 Trybot-Ready: Daniel Jacques <dnj@chromium.org> Reviewed-by: Daniel Jacques <dnj@chromium.org> Commit-Queue: Daniel Jacques <dnj@chromium.org> Tested-by: Daniel Jacques <dnj@chromium.org>
2015-05-16stats: Split out the gauges in the sync stage.Gabe Black
These seem to have been squished together even though they had separate subnames. Instead of relying on subnames, lets give them explicitly distinct gauge stats. BUG=chromium:469860 TEST=Ran run_tests. Change-Id: I6df85f9785dfa6008f2d0c6a95345c2ce968dd80 Reviewed-on: https://chromium-review.googlesource.com/271551 Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Gabe Black <gabeblack@chromium.org> Tested-by: Gabe Black <gabeblack@chromium.org>
2015-05-15Launch multiple trybots together in Pre-CQ launcher.David James
Pre-CQ launcher now submits each tryjob one by one (4 Pre-CQ configs -> 4 tryjobs). This is wasteful. This CL fixes that. BUG=chromium:446554 TEST=Unit tests. TEST=Pre-cq-launcher trybot run. Change-Id: I2ef3f3d31ed308b73b80374dda1c3351bbf63c3a Reviewed-on: https://chromium-review.googlesource.com/268851 Trybot-Ready: David James <davidjames@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> Tested-by: David James <davidjames@chromium.org> Commit-Queue: David James <davidjames@chromium.org>
2015-05-15stats: Add stats for the precq in the sync stage (attempt 2).Gabe Black
These stats measure how many CLs are in various states, and how many tryjobs have been launched. Since a CL being in a particular state is a persistent condition, I'm measuring those with gauges. Launching a trybot is an event whose rate we want to track over time, and that sounds like what a counter is for. BUG=chromium:469860 TEST=Ran run_tests. Change-Id: Idfac735b1b41b403972ed0d890e312ff25ebbca2 Reviewed-on: https://chromium-review.googlesource.com/271324 Reviewed-by: Dan Shi <dshi@chromium.org> Tested-by: Gabe Black <gabeblack@chromium.org> Commit-Queue: Gabe Black <gabeblack@chromium.org>
2015-05-14Revert "stats: Add stats for the precq in the sync stage."David James
This reverts CL:270559, which broke the Pre-CQ launcher. See chromegw/i/chromeos/builders/Pre-CQ%20Launcher/builds/4722 20:48:43: ERROR: <type 'exceptions.AssertionError'>: Traceback (most recent call last): File "/mnt/data/b/cbuild/internal_master/chromite/cbuildbot/failures_lib.py", line 160, in wrapped_functor return functor(*args, **kwargs) File "/mnt/data/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1601, in PerformStage builder_run=self._run) File "/mnt/data/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1330, in AcquirePool if pool.AcquireChanges(gerrit_query, ready_fn, change_filter): File "/mnt/data/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1234, in AcquireChanges self, self.changes, self.non_manifest_changes) File "/mnt/data/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1504, in ProcessChanges status_guage.send(status, count) File "/mnt/data/b/cbuild/internal_master/chromite/lib/graphite_lib/stats.py", line 139, in send statsd.Gauge.send(self, subname, value) File "/usr/local/lib/python2.7/dist-packages/statsd/gauge.py", line 31, in send assert isinstance(value, compat.NUM_TYPES) AssertionError BUG=chromium:469860 TEST=none Change-Id: I11977afa2feb3ba2c239d561bfd23f97c6dc7154 Reviewed-on: https://chromium-review.googlesource.com/271232 Reviewed-by: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2015-05-14stats: Add stats for the precq in the sync stage.Gabe Black
These stats measure how many CLs are in various states, and how many tryjobs have been launched. Since a CL being in a particular state is a persistent condition, I'm measuring those with guages. Launching a trybot is an event whose rate we want to track over time, and that sounds like what a counter is for. BUG=chromium:469860 TEST=Ran run_tests. Change-Id: Ib8b4f5ed712710051dbd4193a86769d61f2aa646 Reviewed-on: https://chromium-review.googlesource.com/270559 Reviewed-by: Dan Shi <dshi@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> Commit-Queue: Gabe Black <gabeblack@chromium.org> Tested-by: Gabe Black <gabeblack@chromium.org>
2015-05-12Deprecate git.ReinterpretPathForChroot().Gilad Arnold
CQ-DEPEND=CL:270493 BUG=brillo:1032 TEST=cbuildbot/run_tests TEST=Trybot Change-Id: I0e48e254af76cedbc9ccb069b4b9800073678516 Reviewed-on: https://chromium-review.googlesource.com/269267 Reviewed-by: Gilad Arnold <garnold@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org>
2015-05-12Deprecate cros_build_lib.{To,From}ChrootPath().Gilad Arnold
BUG=brillo:1032 TEST=cbuildbot/run_tests TEST=Trybot Change-Id: I7403d5af1ddea24ee8caaec648c7c52f83878c73 Reviewed-on: https://chromium-review.googlesource.com/269266 Reviewed-by: Bertrand Simonnet <bsimonnet@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org>
2015-05-11cbuildbot: Mark OverrideConfigForTrybot as Chrome Os specific.Don Garrett
This method is specific to how Chrome Os build configs are generated and used, so restrict it to the Chrome Os specific config library. This doesn't solve the problem, but does call it out more specifically. Eventually, anything which references generate_chromeos_config will clearly be Chrome OS specific and easy to identify. BUG=chromium:483566 TEST=lint + run_tests. Change-Id: Ia4b43ab798fc3f3d625bbb4f1387e885967bec8b Reviewed-on: https://chromium-review.googlesource.com/270112 Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org>
2015-05-11branch_util: Add a argument to branch-util to skip remote push.Prathmesh Prabhu
This CL adds an argument to the branch-util subgroup of cbuildbot to skip pushing to remote repositories completely. This will be used by an integration test that uses branch_util end-to-end, skipping just the push-to-remote part. BUG=chromium:470690 TEST=`cbuildbot --local branch-util --skip-remote-push \ --branch-name blah --version 7041.0.0` Change-Id: I73cb1afdcc1daa9ac7ced916f9b5876e7ba7e06a Reviewed-on: https://chromium-review.googlesource.com/270043 Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
2015-05-09Make beaglebone_servo images build updates for base images.J. Richard Barnette
By default, if a test image is built, the builder creates the update.gz file from that test image. This is the wrong thing for beaglebone_servo: we want the base image, not the test image. This fixes builds to include a new config option for specifying which build should be the source for non-paygen update payload generation. By default, it chooses according to the old rules, but if the option is provided it overrides the default. BUG=None TEST=Unit tests Change-Id: I5f7460a614cf48f7a5cfcf8b1caf764fa9ae31fc Reviewed-on: https://chromium-review.googlesource.com/269836 Commit-Queue: Richard Barnette <jrbarnette@chromium.org> Tested-by: Richard Barnette <jrbarnette@chromium.org> Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
2015-05-08cbuildbot_config: Rename to generate_chromeos_config.Don Garrett
Create minimal cbuildbot_config that imports symbols from gen_cbuildbot_config. This is done to clearly show what external symbols are needed from cbuildbot_config. Eventually cbuildbot_config will read a json file to populate itself, but we need to fully understand the interface to know all of the constraints. BUG=chromium:483566 TEST=run_tests + linting Change-Id: I71cb9d541df2a63efb7342f465418678bd515dc9 Reviewed-on: https://chromium-review.googlesource.com/269953 Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org>
2015-05-07cros lint: check the indentation level of section argumentsMike Frysinger
Sometimes the indentation of things like "Args:" is incorrect. Add a linter check for it and fix up all the existing errors. BUG=chromium:485005 TEST=`cros lint` on all files pass, and catches some bad cases Change-Id: I79ea60d1e07c40d9283284bdaebfcda5af4e6246 Reviewed-on: https://chromium-review.googlesource.com/269642 Trybot-Ready: Mike Frysinger <vapier@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Commit-Queue: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2015-05-07cbuildbot_config: Stop exporting 'config'.Don Garrett
Stop exporting the variable 'config' from cbuildbot_config and instead export 'GetConfig()'. Instead of giving every script access to the global variable 'config', export via a function. This is to simplify upcoming changes that may need to load the config dynamically, or handle not having one. TEST=cbuildbot/run_tests && cros lint *.py BUG=chromium:483566 CQ-DEPEND=CL:269775 Change-Id: Ifaa33a597907cfc18a7e5e99c218609e26eed683 Reviewed-on: https://chromium-review.googlesource.com/269770 Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org>
2015-05-06cbuildbot_config: Get rid of GetManifestVersionsRepoUrl.Don Garrett
The method GetManifestVersionsRepoUrl never really made sense inside cbuildbot_config, since it's not really a config value. So... get rid of the method altogether. It was either being used in places where it had a static result (now replaced with constant values), or via a wrapper in sync_stages. Move the real implementation into sync_stages. TEST=cbuildbot/run_tests && cros lint *.py BUG=chromium:483566 Change-Id: Iddfee41340bfdbd63b6376469e3fbbad18361673 Reviewed-on: https://chromium-review.googlesource.com/269165 Reviewed-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org>
2015-05-05cbuidlbot_config: Rename _config -> BuildConfig.Don Garrett
The _config class is used in a number of unittests so officially export it. Also makes it generally easier to search for. TEST=cbuildbot/run_tests && cros lint *.py BUG=chromium:483566 Change-Id: Ia5a921ff53ab3397927e19cd33de69970f88f1c9 Reviewed-on: https://chromium-review.googlesource.com/268915 Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org>
2015-05-01fix some old users of cros_build_lib outputMike Frysinger
Looks like the fwgdb script was missed in the big migration, as was the sync_stages module (and there looks like a linting bug there). The others are more insidious -- they weren't logging their output, but instead using the builtin exceptions.Warning to construct an object. That's why the linter didn't complain -- the code was valid. BUG=brillo:600 TEST=`cros lint` is happy TEST=`./cbuildbot/run_tests` passes Change-Id: I45207422fbea059bdee70a2eedd120b5688b6350 Reviewed-on: https://chromium-review.googlesource.com/268383 Trybot-Ready: Mike Frysinger <vapier@chromium.org> Reviewed-by: Ralph Nathan <ralphnathan@chromium.org> Commit-Queue: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2015-04-30cbuildbot: only sign factory images for factory configsMike Frysinger
The factory configs are for producing factory install shims only. They also produce recovery images, but we really don't want them: - factory branches have lots of hacks to support factory - they never actually get used anywhere - they're pure overhead to upload/sign Change the pushimage logic to only push factory images when it's a factory config. BUG=chromium:482342 TEST=`./cbuildbot/run_tests` passes TEST=`cbuildbot daisy-factory` runs pushimage just for factory images Change-Id: Id6ad8ec8b25d944991c3b08eb8c16f36424f5097 Reviewed-on: https://chromium-review.googlesource.com/267850 Trybot-Ready: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> Commit-Queue: Mike Frysinger <vapier@chromium.org>
2015-04-30Sync with unpinned manifest first prior to syncing with pinned manifest.David James
Syncing to a pinned manifest ensures that we have the specified revisions, but, unfortunately, repo won't bother to update branches. Sync with an unpinned manifest first to ensure that branches are updated (e.g. in case somebody adds a new branch to a repo.) BUG=chromium:482077 TEST=Confirm that the new repo sync -n is run in a trybot run, and that all repos are updated with new branches. Confirm that repos are not updated with new branches without the fix. Change-Id: I9ed27975eb58c20015d8f8e5d0a6f43baafdd2da Reviewed-on: https://chromium-review.googlesource.com/268365 Reviewed-by: Gabe Black <gabeblack@chromium.org> Commit-Queue: David James <davidjames@chromium.org> Trybot-Ready: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2015-04-30Print chumped changes on CQ slaves again.David James
BUG=chromium:478176 TEST=Test that CQ slaves are now printing chumped changes. Change-Id: I9b6bddbeac940d0fe16f1acc985210be5a8637fd Reviewed-on: https://chromium-review.googlesource.com/268354 Reviewed-by: David James <davidjames@chromium.org> Commit-Queue: David James <davidjames@chromium.org> Trybot-Ready: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-16 00:31:49 +0000