summaryrefslogtreecommitdiff
path: root/scripts
AgeCommit message (Collapse)Author
2012-07-17cros_mark_as_stable: always setup stabilization branch when committingMike Frysinger
We need the stabilization branch in order to commit our updated cache files, so always set that branch up even if we don't have any ebuilds to work on. If there ends up not being any work, and we don't update any cache files, it'll just get thrown away. BUG=None TEST=`./buildbot/run_tests` passes TEST=`cros-cbuildbot --local arm-tegra2-paladin` and look at local branches after it finishes Change-Id: I4e5bece8aae6d59ba894f181266ee81f45db8efb Reviewed-on: https://gerrit.chromium.org/gerrit/27530 Reviewed-by: David James <davidjames@chromium.org> Commit-Ready: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2012-07-17Don't re-raise StepFailure exceptions in cbuildbot main script.David James
StepFailure exceptions are already recorded in the report and in the stage that failed, so there is no need to print them again. BUG=chromium-os:32596 TEST=Run cbuildbot and verify tracebacks are still printed out, but not quite as many times. Change-Id: Id1c5c537de1151d3f26ff28f50306b3cbd7a45de Reviewed-on: https://gerrit.chromium.org/gerrit/27460 Commit-Ready: David James <davidjames@chromium.org> Reviewed-by: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2012-07-13Add --gerrit-chrome option.Ryan Cui
Enables cbuildbot to build Chrome from GERRIT_SOURCE. Contains safety checks so this is not used on remote trybots, since chrome_set_ver can potentially put the repository in a bad state. BUG=chromium-os:32456 TEST=Ran local trybot Change-Id: I6521d5b66df3aa022fa0e51744a9bf5285cf1882 Reviewed-on: https://gerrit.chromium.org/gerrit/27083 Commit-Ready: Ryan Cui <rcui@chromium.org> Reviewed-by: Ryan Cui <rcui@chromium.org> Tested-by: Ryan Cui <rcui@chromium.org>
2012-07-13Move hidden options together.Ryan Cui
BUG=None TEST=Ran cbuildbot Change-Id: Ie88e8845a601a9541c0a281089e7056463d1bcb2 Reviewed-on: https://gerrit.chromium.org/gerrit/27001 Commit-Ready: Ryan Cui <rcui@chromium.org> Reviewed-by: Ryan Cui <rcui@chromium.org> Tested-by: Ryan Cui <rcui@chromium.org>
2012-07-11Make --debug a remote option.Ryan Cui
So that --pass-through='--debug' is not necessary for running debug --buildbot runs on remote trybot. BUG=None TEST=Ran trybot Change-Id: Ic1f88061e70ec874617e41ddc1f4a2d93522d2ad Reviewed-on: https://gerrit.chromium.org/gerrit/27206 Reviewed-by: David James <davidjames@chromium.org> Commit-Ready: Ryan Cui <rcui@chromium.org> Tested-by: Ryan Cui <rcui@chromium.org>
2012-07-11cros_mirror: add missing shift to -j flagMike Frysinger
BUG=None TEST=`cros_mirror -j 13` doesn't error Change-Id: Id0960566554d7da4827e617afb86653b1092113d Reviewed-on: https://gerrit.chromium.org/gerrit/27138 Reviewed-by: Hung-Te Lin <hungte@chromium.org> Commit-Ready: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2012-07-10do not use binpkgs when using a fresh sdkMike Frysinger
When using the freshly created sdk, we do not want to force binpkgs when trying to update chroot packages because the binpkgs don't yet exist -- they're created by this step. Otherwise, we get into the situation where the bootstrapped sdk installs a newer package which is then downgraded when we attempt to use it for the first time. BUG=chromium-os:32354 TEST=`./buildbot/run_tests` passed TEST=`cbuildbot chromiumos-sdk` worked Change-Id: I223c03379f9e34d59e8a48e1ee94a2b3964c57ac Reviewed-on: https://gerrit.chromium.org/gerrit/27028 Reviewed-by: Brian Harring <ferringb@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Zdenek Behan <zbehan@chromium.org> Commit-Ready: Mike Frysinger <vapier@chromium.org>
2012-07-10Call cros_workon from src/scripts.David James
cros_list_modified_packages runs in early phases of the build, so it needs to be robust to cros-devutils not being installed. This is the case for two reasons: 1) If you're in early stages of the chromiumos-sdk builder, cros-devutils won't be installed. 2) If you're on a preflight builder, the uprev stage may uninstall cros-devutils. BUG=chromium-os:31952 TEST=Local run of script. Remote trybot run. Search all repositories for references to CROSUTILS_LIB_DIR (which is removed here). Change-Id: I8beea00530aec47397a5ba11c80b08620a93c9dd Reviewed-on: https://gerrit.chromium.org/gerrit/27044 Tested-by: David James <davidjames@chromium.org> Reviewed-by: Chris Sosa <sosa@chromium.org> Commit-Ready: David James <davidjames@chromium.org>
2012-07-10Delete unused functions and scripts from chromite.David James
The functions deleted by this CL are not currently used, and many of them do not actually work. The following functions are deleted: - cros_build_lib.GetSrcRoot - cros_build_lib.GetOutputImageDir - cros_build_lib.IsProjectInternal The following files are deleted: - {bin,scripts}/cros_build_packages - lib/cros_check_output.py - lib/cros_subprocess.py - lib/text_menu.py - tests/build_image_test.py I've grepped through the source to make sure these files are not referenced anywhere. BUG=chromium-os:32443 TEST=Run all unit tests. Remote trybot runs. Change-Id: Ibdea2aa17fdb9956f59cdb63c77f633429cc3db4 Reviewed-on: https://gerrit.chromium.org/gerrit/27049 Tested-by: David James <davidjames@chromium.org> Reviewed-by: Chris Sosa <sosa@chromium.org> Commit-Ready: David James <davidjames@chromium.org>
2012-07-10Remove deadline for --local flag requirement.Ryan Cui
BUG=chromium-os:32378 TEST=Ran locally Change-Id: Ideb6e07460749f26866f9c3245b1b180dc80dc4e Reviewed-on: https://gerrit.chromium.org/gerrit/26976 Reviewed-by: Brian Harring <ferringb@chromium.org> Commit-Ready: Ryan Cui <rcui@chromium.org> Tested-by: Ryan Cui <rcui@chromium.org>
2012-07-10Track what packages failed and include it in exceptions.David James
It is helpful to track what packages failed so that we can include this information in messages from the commit queue. This will help developers realize which change caused a particular failure. Related refactoring changes: - Rename NonBacktraceBuildException to StepFailure, and move it to cbuildbot_results.py - Rename TestException to TestFailure, and make it inherit from StepFailure, since these exceptions always fail the step, and we don't want to print detailed tracebacks for them. - Rename BackgroundException to BackgroundFailure for consistency with other exceptions in the StepFailure hierarchy. BUG=chromium-os:32249 TEST=Run remote trybot jobs with example build breaks and verify exceptions are correct. Unit tests. Pylint. Change-Id: Icb983797aaf9827288d978dac38118a2c78f7096 Reviewed-on: https://gerrit.chromium.org/gerrit/26744 Commit-Ready: David James <davidjames@chromium.org> Reviewed-by: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2012-07-09Revert "cbuildbot: Use CQ logic for patch resolution/applying."Brian Harring
This reverts commit 11272651bfb62a63320d6bc39e05f590c84bc86e and 5a0de704cd175886e1e98d099c3d016bd58a9f4d . 3 Bugs in total have surfaced; one spotted shortly after, 2 others via remote; 2/3 are fixed, but production isn't the place to sort it out. BUG=None TEST=None Change-Id: I372825097b536ebdf305be200c6a1057a1142a29 Reviewed-on: https://gerrit.chromium.org/gerrit/26995 Reviewed-by: Brian Harring <ferringb@chromium.org> Tested-by: Brian Harring <ferringb@chromium.org>
2012-07-09cbuildbot: Use CQ logic for patch resolution/applying.Brian Harring
Beyond dropping redundancy, this makes testing CQ patch application behaviour much easier- the core chunk of it is directly accessible via cbuildbot interface. In the process this expands cbuildbot support to automatically pull in paladin deps, and gerrit deps. This has the same flaw for bootstrap testing where a CQ-DEPEND that pulls in a chromite change isn't bootstrapped; this is outside this changes purview to fix however. Finally... this is required before we can turn on cherry-picking mode; specifically we need require the gerrit dep resolution angle to fully pull in the necessary parents. Note that this again break the pickle support for ValidationPool against current ToT; this is known (flagged by tests), but is required since the build root needs to be threaded down through the invoking code. BUG=chromium-os:27415,chromium-os:29546 TEST=checkout this patch, and run: cbuildbot x86-generic-full -g I60e1112c849f149f5c5ae0cc1ca6bd752bab72fd TEST=checkout this patch, and run: cbuildbot x86-generic-full -g I4325b5cf8ba0d48b21da9ae9043f95ec917c751f TEST=unittests, pylint, etc. Change-Id: I60e1112c849f149f5c5ae0cc1ca6bd752bab72fd Reviewed-on: https://gerrit.chromium.org/gerrit/20505 Reviewed-by: Brian Harring <ferringb@chromium.org> Tested-by: Brian Harring <ferringb@chromium.org>
2012-07-03Remove dependency map from failures.David James
Dependency maps aren't very useful anymore for debugging failures, so don't show them anymore. This makes the failure output more terse so that the output will focus on the package that failed. BUG=none TEST=Confirm dependency maps aren't printed anymore when parallel_emerge fails. Change-Id: Ic8f02c51380b9f15be0cf4a1e711f8c580b101ff Reviewed-on: https://gerrit.chromium.org/gerrit/26543 Reviewed-by: Mike Frysinger <vapier@chromium.org> Commit-Ready: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2012-07-03Fix chrome_set_ver.py w.r.t. removal of cros.DEPSBen Chan
All dependencies in cros.DEPS have been moved to Chromium's main DEPS. This CL updates chrome_set_ver accordingly. BUG=chromium-os:31867 TEST=cros_workon_make chromeos-chrome Change-Id: I4384e6ae9a85410070bdfe1b4a802830e6b1acd5 Reviewed-on: https://gerrit.chromium.org/gerrit/26614 Reviewed-by: John Sheu <sheu@chromium.org> Tested-by: John Sheu <sheu@chromium.org> Reviewed-by: David James <davidjames@chromium.org> Commit-Ready: Ben Chan <benchan@chromium.org> Tested-by: Ben Chan <benchan@chromium.org>
2012-07-02Add hw test as an option for remote trybotYu-Ju Hong
User can specify "--hwtest" to turn on hw testing for remote trybot. The feature is off by default. This change passes 'trybot-target/version' (e.g. 'trybot-lumpy-paladin/R22-2515.0.0-a1-b127') as an unique label to the autotest server, as opposed to 'target/version' in the buildbot runs. Hence the artifacts are stored at GOOGLE_STORAGE_URL_PREFIX/chrome-image-archive/trybot-target/version/ BUG=chromium-os:32272 TEST=unittest & remote trybot run using --bootstrap-args Change-Id: I07554de1c36e55bac0168b69640082d746bced23 Reviewed-on: https://gerrit.chromium.org/gerrit/26074 Tested-by: Yu-Ju Hong <yjhong@chromium.org> Reviewed-by: Chris Sosa <sosa@chromium.org> Commit-Ready: Yu-Ju Hong <yjhong@chromium.org>
2012-06-30Fix edge case AttributeError.Brian Harring
Occurs if -r target doesn't exist at all. BUG=None TEST=rm -rf $T; cbuildbot <args> -r $T # works. Change-Id: I8fb958bfb47d8e57c4d02f53a2bb3c67a5c618f4 Reviewed-on: https://gerrit.chromium.org/gerrit/26531 Commit-Ready: Brian Harring <ferringb@chromium.org> Tested-by: Brian Harring <ferringb@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org>
2012-06-29cros_mirror: helper for creating local mirrors of ChromiumOS treesMike Frysinger
For organizations with slow connections to our servers, add a script so they can quickly set up local mirrors for developers on the LAN to pull from. This should help with the speed complaints we see from working with our giant repo. Documentation is maintained here: http://dev.chromium.org/chromium-os/how-tos-and-troubleshooting/creating-local-mirrors BUG=None TEST=`./cros_mirror -r /some/path` set up a mirror i could pull from Change-Id: Idef259fa6757e2be787900be89bc71d555eb63f8 Reviewed-on: https://gerrit.chromium.org/gerrit/25263 Tested-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Ulf Hofemeier <ulf.hofemeier@intel.com> Tested-by: Ulf Hofemeier <ulf.hofemeier@intel.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> Commit-Ready: Mike Frysinger <vapier@chromium.org>
2012-06-29Move portage helper functions into portage_utilities.py and add unit tests.David James
BUG=chromium-os:32249 TEST=Run unit tests. Verify cros_list_modified_packages still works with multiple modified packages. Run pylint on all modified files. Change-Id: I439ac2d778fbaf9dffa955a21bdcc69a4bebd375 Reviewed-on: https://gerrit.chromium.org/gerrit/26360 Commit-Ready: David James <davidjames@chromium.org> Reviewed-by: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2012-06-29Allow archive bucket to be specified on the command line.Ryan Cui
This will be used by chromium cros trybots running on the chromium tryserver to specify a different bucket to upload their artifacts to, so they don't conflict with chromeos chromium trybots running on the chromeos tryserver. BUG=chromium-os:31611 TEST=Remote trybot Change-Id: I6f4b9557339af942de322df48b99b3330dd0e470 Reviewed-on: https://gerrit.chromium.org/gerrit/26066 Reviewed-by: Chris Sosa <sosa@chromium.org> Commit-Ready: Ryan Cui <rcui@chromium.org> Tested-by: Ryan Cui <rcui@chromium.org>
2012-06-27portage_utilities: change all cros_workon variables to be bash arraysZdenek Behan
* Ebuilds with a single project are treated exactly like before to avoid a massive change of everything once this is commited * Fixed cros_list_modified_packages to also allow arrays * Fixes all unit tests to match the new behaviour. * Adds another unit test to verify we're able to load both old and new ebuild formats correctly. * Adds a unit test verifying Uprev behaviour of multi-project ebuilds BUG=chromium-os:25338 TEST=unit tests & pylint TEST=ran cros_list_modified_packages with multi and single project ebuilds workon'd TEST=trybot x86-generic-paladin (verify existing ebuilds behaviour) TEST=trybot x86-generic-paladin with experimental glibc patch (project array) Change-Id: Ia09da8d80d574028c4889e493abe1587de2a7f75 Reviewed-on: https://gerrit.chromium.org/gerrit/25518 Tested-by: Zdenek Behan <zbehan@chromium.org> Reviewed-by: David James <davidjames@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Commit-Ready: Zdenek Behan <zbehan@chromium.org>
2012-06-27upload_command_stats: silently move on if host cannot be determined.Matt Tennant
If the cros_build_lib.GetHostName function fails to determine the name of the current host then the upload_command_stats script will now silently skip the upload of stats, rather than crashing. BUG=chromium-os:32171 TEST=`build_packages --board=x86-generic` still uploads for me TEST=Thought traced stack in bug report, high degree of confidence that this fixes the problem. TEST=Locally edited cros_build_lib.GetProjectUserEmail to return None, then verified that new warning gets printed and no stats uploaded. Change-Id: I31154fe8ac89def0f3899a8be62e3c454f9f7eae Reviewed-on: https://gerrit.chromium.org/gerrit/26160 Commit-Ready: Matt Tennant <mtennant@chromium.org> Reviewed-by: Matt Tennant <mtennant@chromium.org> Tested-by: Matt Tennant <mtennant@chromium.org>
2012-06-26Re-land "Add HWTests to Lumpy build config."Chris Sosa
Original change reviewed in I470b42a13e84f66b0d8ae5d0be8c034e45b2d94d. This reverts commit c6b4fcaed4c2bfb22ec4a0821c23db72448b0dd8 with the tweak of just disabling the tests. This gets the funcitonality in without turning it on. BUG=chromium-os:31786 TEST=unittests + local lumpy paladin run + pylint Change-Id: Ie55ca18274381662a5fab58f16763461484dae6c Reviewed-on: https://gerrit.chromium.org/gerrit/26060 Reviewed-by: Scott Zawalski <scottz@chromium.org> Commit-Ready: Chris Sosa <sosa@chromium.org> Tested-by: Chris Sosa <sosa@chromium.org>
2012-06-26upload_command_stats: Switch to standard logging module.Matt Tennant
Runs of upload_command_stats were emitting a DEBUG message from cros_build_lib, because somehow the logging module defaults to the DEBUG level. I switched upload_command_status from using "operation" to using "logging", which also has the effect of setting the logging level to INFO. That disables the debug output that looked weird. BUG=None TEST=`build_packages --board=x86-generic` still uploads, but without the debug message from RunGitcommand. TEST=`upload_command_stats` exits with new error that comes from logging module. TEST=`upload_command_stats` exits with new error that comes from logging module, after propagation from new StatsError exception. Change-Id: I0c6d040d17169bf211e8331e5066fd74ac3dd85b Reviewed-on: https://gerrit.chromium.org/gerrit/26057 Tested-by: Matt Tennant <mtennant@chromium.org> Reviewed-by: Ryan Cui <rcui@chromium.org> Commit-Ready: Matt Tennant <mtennant@chromium.org>
2012-06-26cros_portage_upgrade: check board arch after checking whether board is set up.Matt Tennant
The cros_portage_upgrade script always used to issue an error when a board is not set up before running. However, a new check that uses the arch of each board was inserted before the existing check. Finding a board's arch involves using the board's build directory. This change moves the check which needs a board's arch after the check for whether a board is set up. BUG=chromium-os:29673 TEST=`cros_portage_upgrade --board=link --unstable-ok ws4py` already gives nice error about needing to set up the board. TEST=`ros_portage_upgrade --board=link --unstable-ok --upgrade ws4py` also gives the same error, where before this change it gave a stack. TEST=pylint on both files Change-Id: I3da4afd3ff5cc29bfbef2907725a8659aef912a4 Reviewed-on: https://gerrit.chromium.org/gerrit/26070 Tested-by: Matt Tennant <mtennant@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Commit-Ready: Matt Tennant <mtennant@chromium.org>
2012-06-25Don't check the tree status for Commit Queue's set up on branches.Chris Sosa
This change moves GetChromiteTrackingBranch in to cros_build_lib and uses it to not check tree status when we are not tracking master. BUG=chromium-os:28180 TEST=unittests. Change-Id: Ieb48c46c4e3a2c57ca0027bacaa3b5e2060b8daf Reviewed-on: https://gerrit.chromium.org/gerrit/25533 Tested-by: Chris Sosa <sosa@chromium.org> Commit-Ready: Chris Sosa <sosa@chromium.org> Reviewed-by: Chris Sosa <sosa@chromium.org>
2012-06-25Wrap prebuilt.py using wrapper.py.David James
Right now, the logging isn't set up correctly for prebuilt.py, so all output from prebuilt.py is dropped. This CL sets up the wrapper so that output of prebuilt.py is not dropped anymore. I've also added print_cmd=True, redirect_stdout=True, redirect_stderr=True to calls to gsutil so that we see that gsutil is being called, but we don't get extremely verbose messages about how many MB were copied to Google Storage. BUG=chromium-os:31989 TEST=Run example upload and unit tests. Change-Id: I37e63380ce02639f9c37f0bc52c9f0fd322dcecc Reviewed-on: https://gerrit.chromium.org/gerrit/25687 Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: David James <davidjames@chromium.org> Commit-Ready: David James <davidjames@chromium.org>
2012-06-22Revert "Add HWTests to Lumpy build config."Jon Kliegman
This reverts commit 788566f58603618c359abec61d75657438a24841 Change-Id: I85576f4b3ed92d17e0c7070f0268d38e1a774183 Reviewed-on: https://gerrit.chromium.org/gerrit/25940 Reviewed-by: Jon Kliegman <kliegs@chromium.org> Tested-by: Jon Kliegman <kliegs@chromium.org>
2012-06-19Move STEP_WARNINGS and STEP_FAILURE into constants.py.David James
BUG=none TEST=remote trybot, unit tests. Change-Id: I819c7e7b4e2f85aaf95dc5cc03adab6bdd541bb7 Reviewed-on: https://gerrit.chromium.org/gerrit/25500 Reviewed-by: Brian Harring <ferringb@chromium.org> Commit-Ready: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2012-06-19chromiumos-build-stats: use https and require google.com authMatt Tennant
This configures chromiumos-build-status to use https at all times, and also to require google.com authorization for viewing. BUG=chrome-os-partner:9623 TEST=deployed and tested https as well as login requirement Change-Id: Ib8c0eb43144f62a2c0888b2c4dd754f2aa724b28 Reviewed-on: https://gerrit.chromium.org/gerrit/22423 Tested-by: Matt Tennant <mtennant@chromium.org> Reviewed-by: Chris Sosa <sosa@chromium.org> Commit-Ready: Matt Tennant <mtennant@chromium.org>
2012-06-19cros_setup_toolchains: add an "sdk" target to stop overloading "all"Mike Frysinger
In some places, we're using "all" to mean "all sdk targets", but in others, we're using it to mean "all" targets. Rather than overload the "all" target more, add a new special "sdk" target. BUG=None TEST=`cros_setup_toolchains --show-board-cfg=sdk` shows just sdk targets TEST=`cros_setup_toolchains --show-board-cfg=all` shows all targets Change-Id: I945fa79d09ce58818dc69c9f43b81ab4a0f7baf4 Reviewed-on: https://gerrit.chromium.org/gerrit/25625 Reviewed-by: Zdenek Behan <zbehan@chromium.org> Commit-Ready: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2012-06-19Add HWTests to Lumpy build config.Chris Sosa
This change adds HW Testing to the lumpy CQ build config. For this change, I've sub-classed HW Testing to have a smaller timeout and be forgiving. This is important so that we roll this out slowly without borking the CQ. BUG=chromium-os:31786 TEST=unittests + local lumpy paladin run + pylint Change-Id: I470b42a13e84f66b0d8ae5d0be8c034e45b2d94d Reviewed-on: https://gerrit.chromium.org/gerrit/25575 Tested-by: Chris Sosa <sosa@chromium.org> Commit-Ready: Chris Sosa <sosa@chromium.org> Reviewed-by: Chris Sosa <sosa@chromium.org>
2012-06-19cros_setup_toolchains: split off variant names with --show-board-cfgMike Frysinger
Some tools (like build_image) will call us with the full board name, so handle that edge case. BUG=None TEST=`cros_setup_toolchains --show-board-cfg=tegra2_kaen` works Change-Id: I8f2077e218f4ee0630fdc42180daab3f2f95558a Reviewed-on: https://gerrit.chromium.org/gerrit/25607 Commit-Ready: Mike Frysinger <vapier@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2012-06-17cros_setup_toolchains: support default toolchainsMike Frysinger
Allow toolchain.conf files to mark a tuple as the default. If no default is specified, go with the first one found to match the old behavior. Further, tweak the --show-board-cfg output so that the default toolchain is listed first as some tools need to know that info (so as to deduce target information like ARCH). BUG=None TEST=`cros_setup_toolchains --show-board-cfg=lumpy` now lists x86_64 first rather than i686 Change-Id: Ib0d363a99e3dd31a0271214456fb7d285c908216 Reviewed-on: https://gerrit.chromium.org/gerrit/24454 Commit-Ready: Mike Frysinger <vapier@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2012-06-12cros_setup_toolchains: allow tuples to specify options in toolchain.confMike Frysinger
We have some targets that we don't want to build all the time (for unofficial or experimental boards), so we need to be able to mark them as such. Further, we sometimes need to specify extra options to crossdev when building these targets. Change the toolchain.conf syntax to: <tuple> [json data] This allows us to add as much metadata dynamically as we need. BUG=None TEST=`cbuildbot chromiumos-sdk` works Change-Id: I861fef21cabb0fa12445c96b0e9b83de5e5ba58c Reviewed-on: https://gerrit.chromium.org/gerrit/24335 Reviewed-by: Zdenek Behan <zbehan@chromium.org> Commit-Ready: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2012-06-07Make parallel_emerge logs readable by all.David James
Python does not support creating temporary files that are readable by all, so we need to chmod the file to make it readable. BUG=chromium-os:31194 TEST=Run example merge. Verify the logs are readable by all now. Change-Id: Ibe16838d01f5c9aacb487ab154c8028e0f3666e6 Reviewed-on: https://gerrit.chromium.org/gerrit/24831 Reviewed-by: Mike Frysinger <vapier@chromium.org> Commit-Ready: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org>
2012-06-06Calculate when cros_workon packages are changed.David James
A workon package is treated as changed if any of the below are true: 1) The package is not installed. 2) A file exists in the associated repository which has a newer modification time than the installed package. 3) The source ebuild has a newer modification time than the installed package. This is intended to be used in build_packages to calculate what workon packages to build. It will speed up builds because, if a user runs build_packages, it only rebuilds their workon packages if they made a change, or if a dependency was changed. Some caveats: - We do not look at eclasses. This replicates the existing behavior of the commit queue, which also does not look at eclass changes. - We do not try to fallback to the non-workon package if the local tree is unmodified. This is probably a good thing, since developers who are "working on" a package want to compile it locally. - Portage only stores the time that a package finished building, so we aren't able to detect when users modify source code during builds. BUG=chromium-os:27493 TEST=Verify that packages are listed when their modification times change. Verify that packages are listed when their ebuild changes. Verify that packages are not listed when either of the above is not true. Change-Id: Iec75782fca039fe9da460cbcc5bc2ccc19337c31 Reviewed-on: https://gerrit.chromium.org/gerrit/24210 Commit-Ready: David James <davidjames@chromium.org> Tested-by: David James <davidjames@chromium.org> Reviewed-by: David James <davidjames@chromium.org>
2012-06-04cros_setup_toolchains: support "all" with --show-board-cfgMike Frysinger
We have some tools that want to query all the available tuples, so add support for doing that. BUG=None TEST=`cros_setup_toolchains --show-board-cfg=all` shows all tuples from all boards Change-Id: I4d24f3a8c9bf3d16ade18f25c02a28aaeb50825f Reviewed-on: https://gerrit.chromium.org/gerrit/24332 Reviewed-by: Zdenek Behan <zbehan@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> Commit-Ready: Mike Frysinger <vapier@chromium.org>
2012-06-04cros_setup_toolchains: do not require root for --show-board-cfgMike Frysinger
The --show-board-cfg option just parses config files, so no need to require it to be run as root. BUG=None TEST=`~/trunk/chromite/bin/cros_setup_toolchains --show-board-cfg=x86-generic` works as non-root Change-Id: Ia6b992229a311850fc34886b4243364cfef4771d Reviewed-on: https://gerrit.chromium.org/gerrit/24316 Reviewed-by: Zdenek Behan <zbehan@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> Commit-Ready: Mike Frysinger <vapier@chromium.org>
2012-05-31cros_mark_{chrome_,}as_stable: relocate to scripts/Mike Frysinger
We don't import these as modules, just execute them directly, so move them to the scripts/ subdir. BUG=None TEST=`./buildbot/run_tests` passes TEST=`cbuildbot arm-tegra2-paladin` with manual uprev works TEST=`cbuildbot x86-generic-tot-chrome-pfq-informational` passes Change-Id: I33b064aafa06b506022ba930c953f625bbb13795 Reviewed-on: https://gerrit.chromium.org/gerrit/23569 Commit-Ready: Mike Frysinger <vapier@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2012-05-31cros_sdk: adjust error checking in RemoteTarballExists for multiple layersBernie Thompson
This CL changes the error checking logic in RemoteTarballExists to check for '200 OK' on multiple lines returned in the header. This enables other underlying networking tools to be used, such as proxy services, which happen to output other messages before the curl header output. BUG=None TEST=Manual, verify cros_sdk still runs correctly downloading SDK. Change-Id: I15fdf3c2c8858bc99dc38c993b2b9a0f42081e4a Reviewed-on: https://gerrit.chromium.org/gerrit/23933 Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Bernie Thompson <bhthompson@chromium.org> Tested-by: Bernie Thompson <bhthompson@chromium.org> Commit-Ready: Bernie Thompson <bhthompson@chromium.org>
2012-05-30Standardize to cros_build_lib rather than cros_lib.Brian Harring
While style guides are such that one can use 'as' to rename things, it's a Bad Idea to use varied style in the same codebase like this. If in doubt, look through the patchsets uploaded, logs, etc where we've introduced NameErrors via using the wrong name (is it cros_lib in this module, or cros_build_lib? etc). Conversion/cleanup was done via the following: 1) find chromite -name '*.py' | tee ~/targets | xargs sed -i -e 's:cros_lib:cros_build_lib:g' \ -e 's: import cros_build_lib as cros_lib: import cros_build_lib:' \ 2) pylint --rcfile=chromite/pylint $(<~/targets) 3) <fixing each Line too long>, stopping if none found. 4) $editor $(<~targets) # looking for 'cros_build_lib\.', fixing indentation. 5) Fix any blatant style violations, or bugs spotted while walking the source (intentionally kept to a minimum, but egregious cases where fixed). 6) goto 2. BUG=chromium-os:31384 TEST=buildbot/run_tests TEST=lib/cros_build_lib_unittest.py TEST=pylinting TEST=cbuildbot x86-generic-paladin --remote -p <args> Change-Id: I89f6ab966bbf2cc85a4212bb90057e11ff5d5180 Reviewed-on: https://gerrit.chromium.org/gerrit/24001 Commit-Ready: Brian Harring <ferringb@chromium.org> Reviewed-by: Brian Harring <ferringb@chromium.org> Tested-by: Brian Harring <ferringb@chromium.org>
2012-05-30wrapper: ignore "chromite" in earlier path elementsMike Frysinger
If you happen to have a higher dir named "chromite" in a parent dir where the ChromiumOS repo is checked out, the index('chromite') will locate the wrong one leading to a crash. Change the logic to do a reverse index. BUG=None TEST=move chromiumos/ into a new empty dir named "chromite" and see `cros_sdk` work Change-Id: I56282d31ba8e11aebfc60e77299f1281aa71b538 Reviewed-on: https://gerrit.chromium.org/gerrit/23995 Commit-Ready: Mike Frysinger <vapier@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org>
2012-05-30cros_setup_toolchains: fix splitdebug behaviourZdenek Behan
* Pass FEATURES via crossdev option in order to record it in persistent configuration * Make use of crossdev's reconfigure option to fix configuration of pre-existing targets BUG=chromium-os:31348 TEST=update toolchains, see success Change-Id: I38334744d83836a642cbc872625fcb16b6d7f8aa Reviewed-on: https://gerrit.chromium.org/gerrit/23946 Tested-by: Zdenek Behan <zbehan@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Commit-Ready: Zdenek Behan <zbehan@chromium.org>
2012-05-30sync_package_status: ignore invalid owner values when creating issues.Matt Tennant
The sync_package_status command used by the refresh-packages builder is currently failing because some package rows in the package status spreadsheet have owner values that are not valid Tracker users. For example, it fails on "c-compile-chrome", which is currently the owner on record for the "sys-devel/binutils" package. This change catches that error from Tracker and then handles the new issue as if there is no owner value in the spreadsheet, except with a note in the issue summary about the invalid owner value. BUG=chromium-os:31062 TEST=`sync_package_status --owner=build` runs through, creating bugs like chromium-os:31354. This is the command and row that the builder is currently crapping out on. `python gdata_lib_unittest.py` passes `python sync_package_status_unittest.py` passes pylint of all files passes Change-Id: Ib85193d3da468f0afa4dd9fc457cd9272ca3d133 Reviewed-on: https://gerrit.chromium.org/gerrit/23961 Tested-by: Matt Tennant <mtennant@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Commit-Ready: Matt Tennant <mtennant@chromium.org>
2012-05-25Cleanup git/manifest guts of cros_build_lib, support non-tracking branches.Brian Harring
Purpose of this is to remove redundant implementations, implementations depending on questionable git/repo behaviour, and replace it all with simple verifiable pathways we can have confidence in how it behaves. In the process, the manifest classes are heavily fleshed out to remove redundancy from invoking code, and to simplify invoking code. Finally, note that cbuildbot no longer complains if you tell it to use a branch that isn't tracking another branch. That check has known faults (doesn't work for bisection, doesn't properly account for tracking local branches, etc) and is ultimately unneeded when cherry-picking is in place. BUG=chromium-os:24273 chromium-os:30841 chromium-os:31209 TEST=./run_tests.sh TEST=./cros_build_lib.py TEST=cbuildbot --remote x86-generic-full TEST=cbuildbot --debug --buildbot <chose-your-favorite-paladin> TEST=cbuildbot --buildbot --debug <a-paladin> --version 2274.0.0-rc6 TEST=cbuildbot x86-generic-full --version 2274.0.0-rc6 TEST=cbuildbot arm-tegra2-full --version 2274.0.0-rc6 TEST=get yourself a manifest version syncd repo, run cbuildbot arm-tegra2-full --version 2274.0.0-rc6 TEST=get yourself a manifest version syncd repo, run cbuildbot arm-tegra2-full # aka master. TEST=cbuildbot -p chromiumos/chromite --chrome_version=138412 \ x86-generic-tot-chrome-pfq-informational Change-Id: I15bf62d6fa36e843b380b28b923d872b6ed0aaed Reviewed-on: https://gerrit.chromium.org/gerrit/21934 Reviewed-by: Brian Harring <ferringb@chromium.org> Tested-by: Brian Harring <ferringb@chromium.org> Commit-Ready: Brian Harring <ferringb@chromium.org>
2012-05-25cbuildbot: if --remote --debug, make sure the user knows it.Brian Harring
Basically, this maps back to the fact we're overloading --debug into --dry-run (they *are* different). That needs resolution sooner rather than later. In the interim since I just sat working on other things for the last 3 hours waiting on tryjobs that never were actually submitted due to --debug, modify cbuildbot to output a notice that if --remote --debug was used, jack all was actually done. BUG=None TEST=None Change-Id: I3fef14f37e61fe82115a5c8c4aa14be99cc6ea1e Reviewed-on: https://gerrit.chromium.org/gerrit/23498 Commit-Ready: Brian Harring <ferringb@chromium.org> Tested-by: Brian Harring <ferringb@chromium.org> Reviewed-by: Ryan Cui <rcui@chromium.org>
2012-05-25cros_sdk: Add a --debug option.Brian Harring
This just ups the logging to debug levels if specified. While I'm in here, clean up some unnecessary short option specifications. BUG=None TEST=cros_sdk --debug --enter # makes noise. TEST=cros_sdk --enter # makes less noise. Change-Id: Ic9746a576c1d9ac4a4f71061678f655a8c82e8b1 Reviewed-on: https://gerrit.chromium.org/gerrit/23741 Commit-Ready: Brian Harring <ferringb@chromium.org> Tested-by: Brian Harring <ferringb@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org>
2012-05-25Always re-init to the target manifest branch.Ryan Cui
To allow switching branches, always re-init to the target manifest. The latest versions of repo have full branch switching support. This allows us to have different branch configurations share the same checkout on builders. For production jobs (--debug unset), When there is cleanup failure in a project, delete the .repo/projects/<path> store in addition to the working directory, since a failure may mean the .repo/projects store is corrupted. Saw this when switching between R20 and master when external/skia/third_party/glu was removed from 'master' manifest. This change needs to be integrated into each branch. BUG=chromium-os:21458 TEST=*Ran local trybot switching between branches. *Ran remote trybot switching between R19, R20, and master branches. *Ran alex-paladin with --buildbot --debug. *Tested alex-release, zgb-release, and lumpy-release with manifest-versions-test repo using remote trybots. *Tested that a (manually) corrupted chromite.git was deleted during CleanStage. Change-Id: Ib4620596d2a0f2fe4e98d461a47dec953c54773e Reviewed-on: https://gerrit.chromium.org/gerrit/23175 Tested-by: Ryan Cui <rcui@chromium.org> Reviewed-by: Brian Harring <ferringb@chromium.org> Commit-Ready: Ryan Cui <rcui@chromium.org>
2012-05-24parallel_emerge: Suppress multiprocessing exceptions during shutdown.Brian Harring
Specifically, due to multiprocessing sucking, we wind up seeing the following quite frequently (line wrapped): <parallel-emerge output> Done Exception in thread QueueFeederThread (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 525, in __bootstrap_inner File "/usr/lib64/python2.6/threading.py", line 477, in run File "/usr/lib64/python2.6/multiprocessing/queues.py", line 233, in _feed <type 'exceptions.TypeError'>: 'NoneType' object is not callable Roughly, this (and those like it) come about due to threads still in existance during late vm shutdown. Those threads are around due to multiprocessing leaving those threads to be GC'd while marking the threads as daemon (meaning the VM *can* shutdown regardless of if they're still running). Needless to say, this is stupid. This is fixed in 2.7 via http://bugs.python.org/issue4106. We're not on 2.7 however, and I'm tired of seeing this noisy traceback. This CL basically manipulates the shutdown in a similar fashion, suppressing this fully in my testing (under enough load, it may still be possible for the error to slip out; that would be forkbomb levels of load I suspect). Roughly, addressing this via: 1) Reduced the pointless sys.exit usage; that triggers an Exception, resulting in the vm strongly refererencing the traceback which strongly references each frame it crashed through. One way to keep instances around. 2) Explicitly forcing of gc collection. In testing, this tactic alone seems to be enough to suppress the TB although that's mostly timing. The step is necessary either way to ensure that PyFinalize's GC run doesn't get first crack at these threads (if it does, by that time the VM is mid shutdown thus the exception). 3) Explicitly go looking for those damn threads and try to join them with a 1 second timeout. In testing, this tactic alone seems to be enough to suppress the TB. This is basically a variation of what the upstream fix was. BUG=None TEST=None, realistically. Fundamentally it's a race; heavy system load seems to exacerbate it's frequency, but this can very. Having a build_packages running in the background (full load) while abusing setup_board to create a new target from scratch was the recent 75% scenario- that said that scenario has broken down in the past (right now locally, it's occuring, thus this patch. YMMV however). Change-Id: I2c7c63b23a75808fd33eeefb05d70e73c8463b99 Reviewed-on: https://gerrit.chromium.org/gerrit/23340 Tested-by: Brian Harring <ferringb@chromium.org> Reviewed-by: David James <davidjames@chromium.org> Commit-Ready: Brian Harring <ferringb@chromium.org>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-16 01:28:31 +0000