| Age | Commit message (Collapse) | Author |
|---|
|
On darwin, in repositories with many transitions configured such as
rules_go.git, the action GoToolchainBinaryBuild could be run multiple
times in parallel by Bazel.
Each of these actions would execute a command like this in a sandbox:
```
$GO build -o $OUTPUT -trimpath \
go/tools/builders/ar.go \
go/tools/builders/asm.go \
go/tools/builders/builder.go \
go/tools/builders/cgo2.go \
go/tools/builders/compilepkg.go \
go/tools/builders/cover.go \
go/tools/builders/edit.go \
go/tools/builders/embedcfg.go \
go/tools/builders/env.go \
go/tools/builders/filter.go \
go/tools/builders/filter_buildid.go \
go/tools/builders/flags.go \
go/tools/builders/generate_nogo_main.go \
go/tools/builders/generate_test_main.go \
go/tools/builders/importcfg.go \
go/tools/builders/link.go \
go/tools/builders/pack.go \
go/tools/builders/read.go \
go/tools/builders/replicate.go \
go/tools/builders/stdlib.go \
go/tools/builders/stdliblist.go \
go/tools/builders/path.go
```
Then, each `go build` process would execute multiple `go tools compile`
command up to the number of max Go routines it is set to. The default
max number of Go routines for each `go build` process is the number of
core on the machine executing the action.
On a local build, this means that we could have up to n x n number of
goroutines running in parallel when combining the asynchronousity of
both Bazel and `go build`.
After upgrading to Go 1.20.x, we observed that such parallelism could
get a local MacOS machine run into a deadlock on the OS-level, halting
spawn of new processes such as `git config`, or new Chrome tab browser.
Typical sampling of a `go build` process stuck in deadlock would result
in the following call graphs:
```
2430 Thread_44978 DispatchQueue_1: com.apple.main-thread (serial)
+ 2430 ??? (in <unknown binary>) [0x1358]
+ 2430 runtime.asmcgocall.abi0 (in go) + 124 [0x100b702ac]
+ 2430 runtime.syscall6.abi0 (in go) + 56 [0x100b71a98]
+ 2430 __wait4_nocancel (in libsystem_kernel.dylib) + 8 [0x1a17a04f4]
2430 Thread_44999
+ 2430 runtime.asmcgocall.abi0 (in go) + 201 [0x100b702f9]
+ 2430 runtime.pthread_cond_timedwait_relative_np_trampoline.abi0 (in go) + 28 [0x100b717ec]
+ 2430 _pthread_cond_wait (in libsystem_pthread.dylib) + 1276 [0x1a17d45a0]
+ 2430 __psynch_cvwait (in libsystem_kernel.dylib) + 8 [0x1a1797710]
2430 Thread_45038
+ 2430 runtime.kevent_trampoline.abi0 (in go) + 40 [0x100b71518]
+ 2430 kevent (in libsystem_kernel.dylib) + 8 [0x1a179a060]
```
We could not reproduce outside of Bazel (using bash & to parallelize `go
build`). We also could not reproduce this issue with Go 1.19 releases.
Let's apply a rudimentry workaround by limitting the concurrency of `go
build` process by setting it's max Go routines to 1. This should make
this and future race conditions happen a lot less on all platforms.
By doing this, we rely more on Bazel and less on Go runtime to saturate
our CPU resources.
This change should not affect remote execution as the default
ResourceSet for each action is set at 1 CPU and 250MB RAM.
|
|
Using the new `RunEnvironmentInfo` provider in Bazel 5.3.0, `go_binary`
targets can now specify environment variables to be set when they are
executed with `bazel run`.
Since `RunEnvironmentInfo` generalizes the `testing.TestEnvironment`
provider, the existing usage is updated.
|
|
* always produce .a files at the beginning of a build
This change updates rules_go in anticipation for Go 1.20, which will
ship without .a files for the standard library. Now, instead of
conditionally building the standard library if the configuration
differs from the default, the standard library will always be built.
In addition, the builder will be built using 'go build' instead of
individual invocations of go tool compile/link to avoid needing to
bootstrap the standard library before producing the builder.
* fix a bad cleanup in stdlib.bzl
* redefine transition in terms of what's kept
* use go_reset_target for the builder
This allows the builder to be built in the same configuration
as other targets from
//tests/intagration/reproducibility:reproducibility_tests, so the
bazel-bin directory doesn't get deleted as it would in a mulit-config
build.
* remove previosly deleted code that came back in a merge`
* fix buildifier issue
* try to use tmpdir on WINDOWS
* try another way of getting gocache dirs for windows
* add output
* debug
* debug2
* debug 3
* debug 4
* debug 5
* debug 6
* only compile stdlib .a files for Go 1.20+
* set version to 1.20rc1 for CI testing purposes
* fix libs attribute of Go sdk to allow zero libs files
* fix coverage test
* get only digits of the part before the next dot
* allow empty
* set GOEXPERIMENT=nocoverageredesign
* conditionally turn off goexperiment based on versioun
* rename variable
* fix flag
* add lcov fix
* address comments, and remove one config of builder
* decide to compile .a files based on whether they exist
* fix add call
* add nocoverageredesign to experiments earlier
* another small fix
* fix double
* Update go/private/sdk.bzl
Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
* add a simple test doing a build on 1.20rc1
Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
|
|
As per https://github.com/bazelbuild/bazel/issues/15157 currently we
can't configure any transition-relevant attributes using select. This is
because we use self-transitions on targets, and when this happens,
configurable attributes aren't passed to the transition, so we make
incorrect transition decisions.
We only actually consult configuration data in `_go_context_data`, so we
only actually need to transition on the edges which (transitively) reach
a `_go_context_data`, which is `_go_context_data` itself and `deps`.
|
|
When linking a go_binary in one of the native archive modes, all
native libraries of transitive dependencies should be included in the
resulting CcInfo, not just the libraries obtained from the cdeps.
This commit adds a `cc_info` field to the `GoSource` provider that
includes the result of merging the `CcInfo`s of all `deps` and `cdeps`
for all Go libraries, not just for those using cgo themselves.
|
|
* Remove unnecessary rpath special handling on macOS
Based on
https://github.com/bazelbuild/bazel/issues/10254#issuecomment-1125218345
and https://github.com/bazelbuild/bazel/pull/12304 being fixed, this
special handling of rpaths on macOS appears to be unnecessary.
This cleanup ensures that Bazel sets the correct metadata for the exec
location of Go libraries linked in c-shared mode, which in turn allows
to not include them in the runfiles of all dependents - cc_* targets
depending on them will now generate the correct rpath entries to find
the libraries at runtime at the usual position.
* Don't include non-executable go_binary in dependent's runfiles
If a go_binary is built with a non-executable link mode such as
`c-archive`, its dependents currently pick up a runfile dependency on it
since its DefaultInfo specifies the resulting archive as an executable.
This is unnecessary as the dependent should be able to decide whether to
include the file (e.g. dynamic linking) or not (e.g. static linking).
With this commit, the executable field of the DefaultInfo is only
populated if the go_binary is built with an executable link mode.
Follow-up to #3143
|
|
* Use repo-relative labels in load statements
* Introduce a constant for the Go toolchain requirement
Gets rid of a common source of repo-absolute labels. Repo-relative
labels for toolchains aren't supported with Bazel 4. Now that
config_common.toolchain_type exists, the string label may also be
replaced with a more complex Starlark type in the future.
* Use Label(...) in macros
This resolves a label literal relative to the call site of the Label
constructor.
* Use repo-relative labels in rule attribute defaults
|
|
* Trim transitioned Go settings on non-Go dependencies
By resetting Go-specific settings changed by go_transition to their
previous values when crossing a non-deps dependency edge, e.g. srcs or
data, dependencies through those edges are not affected by the change in
Go settings. This has two advantages:
* Correctness: Previously, if a go_binary with linkmode = "c-archive"
used another go_binary to generate srcs, that go_binary would also be
built as a c-archive and thus fail to run during the build. This was
observed in https://github.com/bazelbuild/bazel/issues/14626.
* Performance: With the new Bazel flag
--experimental_output_directory_naming_scheme=diff_against_baseline,
transitions can return to the top-level configuration since the
Starlark settings that were affected by a transition at some point
during the build are no longer tracked explicitly in a setting, but
computed as a configuration diff. Resetting the configuration for non-
deps dependencies thus prevents redundant rebuilds of non-Go deps
caused by changes in Go settings, achieving a version of
"configuration trimming" for Go.
* Verify transition hermeticity for go_{binary,library,test}
The test verifies that Go settings such as gotags and linkmode do not
affect dependencies through attributes other than deps.
|
|
If a go_binary's `linkmode` is neither `normal` nor `pie`, the resulting
file can't be executed. In this case, `executable` should be set to
`False` on the go_binary rule so that Bazel doesn't create a runfiles
tree for the targets and does not include e.g. a resulting static
library in the runfiles of dependants.
|
|
A CGo-generated header file from any repository, external or main, must
always be includeable via its path relative to its repository root.
Previously, this was not true for CGo targets in external repositories,
which required an additional `external/<repo name>/` prefix, thus
breaking references within the external repository.
Fixes #3109
|
|
* fix minor typos
* update the docs file as the code changes
|
|
Avoids needing to manually keep rst content in sync with starlark
changes.
|
|
Up until now, `ios` was supported as `darwin_arm64` by Go. `GOOS=ios`
support was added in Go 1.16 [1].
1. https://golang.org/doc/go1.16
Signed-off-by: Steeve Morin <steeve@zen.ly>
|
|
The goroot is encoded in the output binary, but the goroot is different
in different workspaces. As a result tool binaries (such as builder) are
not reproducible. Bazel will have cache misses for build involving
builder even if the resulting output from the build is repoducible.
Similar go_binary builds, suppress the goroot path.
|
|
Previously, we assumed that any dynamically linked libraries are in
execroot. This may not be the case if the library is NOT generated.
However, the library should always be in runfiles.
Fixes #2867
|
|
This PR makes go_binary propagate CcInfo.
Fixes #2433
|
|
go_library, go_binary, and go_test now support the //go:embed
directive in Go 1.16. When building a package containing a //go:embed
directive, they'll match patterns against embedable files to generate
an embedcfg file to pass to the compiler.
Embeddable files must be listed in the new embedsrcs attribute.
The GoSource provider now has an embedsrcs field to support this.
Fixes #2775
|
|
There are a few minor manual modifications that needed to be performed.
For example, some .bzl files depend on .bzl files that are generated
outside of this repo and do not presently exist. @bazel_tools is a
great example of this.
Additionally this effort has revealed that bazel gazelle has an explicit
dependency on the private parts of rules_go in
internal/gazelle_binary.bzl on line 21 where it imports from
go/private:rules/transitions.bzl which is no longer the path to import
from once I created the BUILD file in go/private/rules.
As a temporary fix for this, I'm included a patch that changes that to
point to the new path. I will provide a follow up patch once rules_go
has been released to fix it properly in the gazelle repo and then remove
the patch from rules_go.
Fixed: #2619
Bug: bazelbuild/bazel-skylib#250
Bug: bazelbuild/bazel-gazelle#803
|
|
run_shell requires bash which is not available on Windows
without a msys installation.
The shell was only used for "&&" anyway. We can just use
two separate actions.
-trimpath requires an absolute path. On Windows we build
a batch file to expand %cd%. On Unixes we can keep using
run_shell.
|
|
This changes how the goos, goarch, race, msan, static, and pure
attributes of go_binary and go_test are implemented.
Previously, the go_binary and go_test rules used an aspect on the deps
and embed attributes. If any of the mode attributes listed above were
set, the aspect declared additional output files and actions to
generate them for the specified configuration. This approach had two
significant problems. First, the aspect caused the analysis to be
performed twice, wasting time and memory even in the case
where no cross-compilation was needed. Second, the aspect was not
compatible with conditional dependencies specified with select
expressions: aspects run on the configured target graph, but the mode
attributes could not affect Bazel's configuration.
This change deletes the aspect and implements these attributes using a
different mechanism: build settings and configuration transitions.
A new package is added //go/config with a build setting for each
attribute. Most settings are flags that may be set on the
command-line. This is an alternative to --feature flags, which will be
deprecated and eventually removed.
The target //:go_config gathers build settings relevant to Go and
provides a GoContextInfo (private).
The target //:go_context_data depends on //:go_config, //:stdlib,
//:nogo, and a few other standard dependencies. go_binary and go_test
rules no longer need to depend on these targets directly. Go rules
should now only need to depend on //:go_context_data and the Go
toolchain. Unfortunately, neither //:go_context_data nor //:go_config
can be part of the toolchain because the toolchain is always analyzed
for the execution configuration, not the target configuration.
Because of the simplified dependencies, the go_rule function is no
longer needed by most rules. This change makes it possible to
deprecate it.
Fixes #1351
Fixes #2219
Updates #2302
|
|
//tests:buildifier_test checks for lint errors in WORKSPACE, build
files, and .bzl files. It checks for the default set of warnings,
minus a few exclusions.
|
|
* Upgrade gazelle past go1.13 incompatibility.
* go_test: set testing flags after calling testmain.MainStart for
compliance with golang.org/cl/173722.
* go_binary: don't set package name to "main" in plugin and shared modes.
* Disable errorsas test in tools that requires 'go list'. This test
was disabled before 1.13.
* Disable ipv4 test due to "package height" error triggered by
external test depending on both internal package and package under test.
|
|
Go 1.13 is stricter on the package path passed to go tool compile -p.
This means that go_library passed in go_binary.embed fails to declare a
main.main function, as they now retain their original import path. Fix
that by adding a is_main boolean to the GoLibrary provider, which will
tell wether a library is supposed to be a main package; that is,
compiled from within a go_binary.
Fixes #2133
|
|
Objective C and C++ sources may now be compiled directly with
go_library, go_binary, and go_test without needing to use the cgo
legacy wrapper. The cpp toolchain must provide a compiler thats
supports these languages.
Appropriate flags will be passed to the compiler for each language,
but no additional objcopts, objcxxopts attributes are provided. These
are meant to accomodate CGO_CFLAGS, CGO_CXXFLAGS, and there is are no
CGO_MFLAGS, CGO_MMFLAGS directives. These attributes can be added if
necessary though.
Updates #1957
|
|
Files loaded from //go:deps.bzl cannot depend on @bazel_skylib, so
they'll continue using the vendored copy. They only need versions.bzl,
which doesn't have compatibility problems.
Files loaded from //go:def.bzl should depend on @bazel_skylib.
Fixes #2071
|
|
GoCompilePkg will be a replacement for GoCompile and several other
actions. It will consolidate all the work of building a Go archive
into a single Bazel action: it will run the Go compiler, the C/C++
compilers, the assembler, cgo, coverage, nogo, and pack all as one
action. This should reduce sandboxing overhead for local
configurations and file transfer overhead for remote configurations.
This change introduces a very limited version of GoCompilePkg with
most of the above functionality marked "TODO". It can only run the Go
compiler. emit_archive uses this action for archives that don't
require any other features.
Updates #1957
|
|
Fixes #2010
Signed-off-by: John Millikin <jmillikin@stripe.com>
|
|
Set GOROOT explicitly when invoking "go tool compile" and
"go tool binary". The Go tool determines GOROOT automatically based on
argv[0], but this does not work in some sandbox environments.
Fixes #2005
|
|
This change consolidates most actions into a single binary `@go_sdk//:builder`, which is a private dependency of the toolchain instead of an implicit dependency of rules. The public API has not changed. This has several benefits:
* Better abstraction in the toolchain; alternative toolchains can use their own builders.
* Fewer binaries need to be built and sent to the remote cache.
* Actions can share code easily, which will enable further consolidation of actions.
The following actions were converted:
* GoStdLib
* GoAsm
* GoCover
* GoPack
* GoCompile
* CGoCodeGen and CGoImportGen
* GoLink
* GoTestGenTest
* GoGenNogo
Fixes #1891
|
|
This behavior matches the native C++ and Java rules[1] and allows users
to easily verify a target compiles without invoking the linker
unnecessarily.
[1]: https://github.com/bazelbuild/bazel/commit/1dcd8f0f12bb582cc449381f8245f943121031fb
|
|
This removes the need for --incompatible_disallow_dict_plus
|
|
The "data" configuration has been deprecated for a while and has no
effect.
|
|
|
|
Fix Skylark deprecation warnings
Fixed several instances of deprecated Skylark behavior
* Used depset.to_list() when iterating
* Avoid depset + operator
* Remove instances of FileType
* Use native.repository_name() and package_name instead of
REPOSITORY_NAME and PACKAGE_NAME.
Also, upgraded vendored Skylib to 0.4.0, since it had some deprecated
stuff.
This change does not fix deprecated uses of args.add. That will
require more work.
Fixes #1534
Fixes #1535
|
|
* Removed support for Go 1.8.
* Removed go_prefix and everything that mentioned it. importpath is
now required for go_library and go_proto_library. It can be
inherited from embedded libraries though.
* Removed legacy go_repository and its dependencies. @bazel_gazelle is
still needed for the gazelle rule.
* Updated deprecation schedule.
* Deleted obsolete go/DESIGN.md.
* Deleted obsolete go/deprecated.rst.
|
|
Fixes #1458
|
|
go_binary and go_test once again support the importpath
attribute. This is used by go_path to place binary and test sources in
the output tree. It may also be used by coverage for reporting sources
in a GOPATH-like format (though test sources are normally excluded
from coverage).
|
|
These features have been printing deprecation warnings for a while and
are scheduled for removal in 0.12.0.
* library attribute in go_library, go_binary, go_test.
* linkstamp attribute in go_binary, go_test.
* new_go_repository (but not rules_go go_repository yet).
* go_sdk and go_repositories.
* cgo_library and cgo_genrule
* go_proto_library.bzl
Fixes #1208, #812
|
|
* Files in cdeps ending with .dll, .dylib, or .so are considered
dynamic libraries, regardless of target platform (Bazel prefers to
build .so files on all platforms).
* The link action adds rpaths for dynamic libraries provided through
cdeps, using $ORIGIN and @loader_path on Linux and macOS,
respectively. Not sure what the equivalent on Windows is yet.
* The binary method now builds and returns runfiles, which the binary
and test rules return directly. These include data runfiles and
dynamic libraries.
Fixes #1435
|
|
go_binary now has an "out" attribute which allows users to set the
output filename for the generated executable. When set, go_binary will
write this file without mode-specific directory prefixes, without
linkmode-specific prefixes, and without platform-specific extensions.
Fixes #1239
|
|
Also:
* Strip archive metadata when using c-archive mode
* Use extldflags from the Go builder environment
* Build stdlib in shared mode when linking in c-shared mode
* Add pthread to linkopts in c-archive mode
* Compile binaries with -shared when using c-shared linking mode
* Add =all slug to gc/ld/asm flags starting at Go 1.10
* Put generated header in its own directory
* Put clinkmodes setup in cgo file, make setup targets private
* Do not automatically include cgo exports directory headers
|
|
Passing bootstrap = True to go_rule now only controls whether _stdlib
and _builders attributes are added. stdlib continues to do this.
The presence or absence of _builders is no longer used to force off
race mode. This prevented stdlib from being built in race mode without
the aspect. Now, go_tool_binary (the only other bootstrap rule) has a
special _race = "off" attribute, which get_mode looks for.
Fixes #1316
|
|
* Propagate linkmode into the binary rules
* Review fixes
|
|
I am having to change the hidden defaults again, and it's made me realise it's
an ugly thing to make many places do, so I am hiding it in a wrapper and
documenting it as the public interface for writing a rule that uses go_context
|
|
* don't use stdlib at all for bootstrap rules
They just use the underlying sdk directly.
Also, we only ever use the original copy of the main go executable now.
* Renamed while documenting
|
|
The command I ran:
```shell
$ find . \( -name '*.bzl' -or -name '*.BUILD*' -or -name 'WORKSPACE' \) -exec buildifier {} +
```
additional changes and fixes
* implementation function should be on top in rule definitions
* no newline between definitions and doc strings
* Skylib is vendored. No modification allowed.
* Keep entries in SDK_REPOSITORIES lined up
|
|
* Make importpath optional on go_test and go_binary
This allows rules to infer their import path from their embeds, and removes the
need for it on things that build binaries.
* Review fixes
|
|
* Support transitive stamping
go_library rules now support the x_def attribute.
The GoSource provider carries these to the archive.
The GoArchive provider carries the x_def values from it's GoSource as well as
from the full transitive set of archives it depends on.
If an x_def entry has no package it is given the importpath of the library the
GoSource was built from.
Fixes #628
* Add some doucmentatin for stamping
|
|
* Add GoContext
This collects up all dependancies that were on the toolchain, but shoudld be in
target configuration (rather than host) and migrates them to a common
go_context_data dependancy.
It then adds a GoContext object, that holds the toolchain, build mode, standard
library and all mode specific attributes. go_context(ctx) will build one of
these and prepare to use it.
This simplifies a lot of the logic in the rules, and helps enforce the isolation
of action methods from ctx attributes.
* Review feedback plus documentation
|
|
* Fix aspect based proto builds
This changes the structure of our providers.
It is now much more principled about configuration specific and transitive
providers, separating them cleanly.
It also allows the GoLibray provider to supply a function that can be used to
produce the configuration specific version (GoSource). This allows us to fix the
proto library dependancies when built in aspect mode.
* Make it a little easier to write simple code generation rules
* Review fixes
|
