Limit the resources Bazel is allowed to use during the build to 1/2 the
available RAM and 3/4 the available CPU cores. This should help avoid
overwhelming the build machine.
Preserving existing behavior: the bash completion was not executable,
the zsh completion was; according to lukegb the fish completion does
not have to be executable.
flat hashes can be substituted through hashed-mirrors, while recursive
hashes can’t. This is especially important for Bazel since the bazel
fetch dependencies can come from multiple different methods (git,
http, ftp, etc.). To do this, we create tar archives from the
output/external directory, which is then extracted to build. All of
the Bazel hashes are all updated.
Using `bazel_self` for self-references makes managing bazel versions
easier: their less risk of changing defaults or copy pasted code for no
versions leading to incorrect self-references.
From Bazel 2.0.0 onwards, Bazel looks for a binary named
`bazel-${version}-${os_arch}` if the project root contains a
`.bazelversion` file or the USE_BAZEL_VERSION environment
variable is set.
This change ensures we output a binary with the correct name
for the current version and OS/arch combination.
- Bazel strict action env set a default PATH to
`/bin/:/usr/bin:/usr/local/bin`. This was previously changed to disable
this behavior to improve hermeticity. However the previous change was
only removing `/bin:/usr/bin`, keeping `/usr/local/bin`, this commit
also remove this entry.
- Upgraded dependencies
- dependencies script upgraded to take into account new WORKSPACE
rules
- Tests now depends on the `distdir`
Runtime bazel now also depends on the `distdir` setting which appears
in the global configuration file. This increases the bazel closure
size by 85 MO for stuffs which can normally be downloaded at runtime
by bazel. However, any invocation of `buildBazelPackage` (such as in
`bazel-watcher`) may fail in nix sandbox if theses files are not
available at runtime.
If this overhead is too important, we may later evolve to a finer
grained solution, where buildBazelPackage declares the list of
necessary dependencies.
The bazel build patches paths like `/usr/bin/install_name_tool` to refer
to `${cctools}/bin/install_name_tool` instead. If the corresponding
runtime dependency is not denoted, then darwin users can encounter "file
not found" errors, e.g. when they fetch bazel from a binary cache and
don't have `cctools` in their own nix store.
The shebang in the python stub template was incorrectly patched to
```
<store-path>/bin/env python
```
instead of
```
<store-path>/bin/python
```
The reason was that `patchShebangs` was called with `--replace` which is
an unknown argument.
```
patching script interpreter paths in src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt --replace /usr/bin/env python /nix/store/w7gsq8v86hni4ynaqgwwlnlny115ylng-python3-3.7.4/bin/python
find: unknown predicate `--replace'
```
Using `substituteInPlace` instead resolves that issue.
The wrong shebang caused failures of `py_binary` targets due to `python`
not being in `PATH` in certain circumstances.
Turns out markers are non-deterministic after all and even our patching still
doesn't solve this problem completely. For example (tensorflow deps, this is a
complete diff so actual dependencies don't differ):
30509c30509
< bc527ff00916b15caee38793bca8f294c748df4a256de55c5199281be0489e73 result/@bazel_skylib.marker
---
> 4e0303e815c78df1e43d4b88dfe65e73046e0c6157fb10aa9a4e8b910113cd9c result/@bazel_skylib.marker
31045c31045
< fa13d04b2316214c3b4008b52546c2d5b633e006f6f019d597bb3f9745bacf7b result/@bazel_toolchains.marker
---
> b36174bf5535e5157801b6de30c35ee03a03fe57766306393c3d65dd65cbebf4 result/@bazel_toolchains.marker
31144c31144
< b0ce4a3ac29ac22528336dd3a54b5b7af9ecc43bef2a2630713c1981a5cbbb51 result/@build_bazel_rules_swift.marker
---
> 7492528068ec4f8e7ace2ecf8f933ec4e1b2235bd7426ce6f70177919f1cd05e result/@build_bazel_rules_swift.marker
36245c36245
< be2993536a8233d63251b664caf35b1e7cd57d194ab2a39a293876c232d6bbd0 result/@io_bazel_rules_closure.marker
---
> b6655cc3f2c78525e5a724d8a4e93b1e7f09f1e09fc817d231109e7f39103e88 result/@io_bazel_rules_closure.marker
36329c36329
< 087bc674c9509dfe157400d111db4a13eeb45fc76aeccd490cee9aad6771ecad result/@io_bazel_rules_docker.marker
---
> f920ec07315ec71e800b05cd22b2a341c0a80807c6e335ee81739b13c532b422 result/@io_bazel_rules_docker.marker
79544d79543
< 85893a05a817036c61f6cd9f8247757baa1654f473c494ce4fc5253c2bbd2790 result/@platforms.marker
And here's an example of differences:
$ cat result-a/@bazel_skylib.marker
7dc7472d37424ba5ec6a5532765bc911
$MANAGED
cat result-b/@bazel_skylib.marker
a8f3f577798201157128e8e9934c4705
$MANAGED
Instead of trying to patch these markers further we now completely clear them.
Nix hacks for ignoring markers are restored and expanded so that we don't even
attempt to parse the marker.
Also drop preInstall cleanup for dependencies. The reason is while it's more
thorough than default cleanup in buildBazelPackage if such a problem happens we
should fix buildBazelPackage instead. Perhaps even move this (awesome!) snippet
there but it's very slow-running so we'd rather attempt to fix it in other
ways.
Anyway after an update at least .deps build, checked with `nix-build -A --check`.
Timestamp verification skip is no longer needed (not sure why). Generally we
better off always using the environment hack for all packages because that
ensures all NIX_* flags are correctly applied.
One possible improvement in future is to filter only NIX_* variables to
passthru in Bazel.
Without this Bazel always picks Python 3 which breaks Python 2 packages.
Strangely enough just dropping this patch works, with all `bazel.tests`
passing.
Without this Bazel always picks Python 3 which breaks Python 2 packages.
Strangely enough just dropping this patch works, with all `bazel.tests`
passing.
--distdir is now used in the installCheckPhase for prefetched
repositories. That's simpler, more robust and easier to extend in
the future.
Note that `name` argument of fetchurl was removed because it changed
the basename of the generated file and bazel uses this basename for
its cache behavior.
On Darwin, the last argument to GCC is coming up as an empty string.
This is breaking the build of proto_library targets. However, I was not
able to reproduce with the example cpp project[0].
This commit patches the cc_wrapper of Bazel that gets installed on
Darwin to remove the last argument if it's an empty string. This is
not a probem on Linux.
[0]: https://github.com/bazelbuild/examples/tree/master/cpp-tutorial/stage3
- Fixs for newly introduced bin/bash hardcoded reference
- Bazel now references `remote_java_tools_xxx` which contains prebuilt
binaries. We prefetch them, fix them, and force bazel to use the
fixed repository.
It also closes#63096
