- Upgraded dependencies
- dependencies script upgraded to take into account new WORKSPACE
rules
- Tests now depends on the `distdir`
Runtime bazel now also depends on the `distdir` setting which appears
in the global configuration file. This increases the bazel closure
size by 85 MO for stuffs which can normally be downloaded at runtime
by bazel. However, any invocation of `buildBazelPackage` (such as in
`bazel-watcher`) may fail in nix sandbox if theses files are not
available at runtime.
If this overhead is too important, we may later evolve to a finer
grained solution, where buildBazelPackage declares the list of
necessary dependencies.
The bazel build patches paths like `/usr/bin/install_name_tool` to refer
to `${cctools}/bin/install_name_tool` instead. If the corresponding
runtime dependency is not denoted, then darwin users can encounter "file
not found" errors, e.g. when they fetch bazel from a binary cache and
don't have `cctools` in their own nix store.
The shebang in the python stub template was incorrectly patched to
```
<store-path>/bin/env python
```
instead of
```
<store-path>/bin/python
```
The reason was that `patchShebangs` was called with `--replace` which is
an unknown argument.
```
patching script interpreter paths in src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt --replace /usr/bin/env python /nix/store/w7gsq8v86hni4ynaqgwwlnlny115ylng-python3-3.7.4/bin/python
find: unknown predicate `--replace'
```
Using `substituteInPlace` instead resolves that issue.
The wrong shebang caused failures of `py_binary` targets due to `python`
not being in `PATH` in certain circumstances.
Turns out markers are non-deterministic after all and even our patching still
doesn't solve this problem completely. For example (tensorflow deps, this is a
complete diff so actual dependencies don't differ):
30509c30509
< bc527ff00916b15caee38793bca8f294c748df4a256de55c5199281be0489e73 result/@bazel_skylib.marker
---
> 4e0303e815c78df1e43d4b88dfe65e73046e0c6157fb10aa9a4e8b910113cd9c result/@bazel_skylib.marker
31045c31045
< fa13d04b2316214c3b4008b52546c2d5b633e006f6f019d597bb3f9745bacf7b result/@bazel_toolchains.marker
---
> b36174bf5535e5157801b6de30c35ee03a03fe57766306393c3d65dd65cbebf4 result/@bazel_toolchains.marker
31144c31144
< b0ce4a3ac29ac22528336dd3a54b5b7af9ecc43bef2a2630713c1981a5cbbb51 result/@build_bazel_rules_swift.marker
---
> 7492528068ec4f8e7ace2ecf8f933ec4e1b2235bd7426ce6f70177919f1cd05e result/@build_bazel_rules_swift.marker
36245c36245
< be2993536a8233d63251b664caf35b1e7cd57d194ab2a39a293876c232d6bbd0 result/@io_bazel_rules_closure.marker
---
> b6655cc3f2c78525e5a724d8a4e93b1e7f09f1e09fc817d231109e7f39103e88 result/@io_bazel_rules_closure.marker
36329c36329
< 087bc674c9509dfe157400d111db4a13eeb45fc76aeccd490cee9aad6771ecad result/@io_bazel_rules_docker.marker
---
> f920ec07315ec71e800b05cd22b2a341c0a80807c6e335ee81739b13c532b422 result/@io_bazel_rules_docker.marker
79544d79543
< 85893a05a817036c61f6cd9f8247757baa1654f473c494ce4fc5253c2bbd2790 result/@platforms.marker
And here's an example of differences:
$ cat result-a/@bazel_skylib.marker
7dc7472d37424ba5ec6a5532765bc911
$MANAGED
cat result-b/@bazel_skylib.marker
a8f3f577798201157128e8e9934c4705
$MANAGED
Instead of trying to patch these markers further we now completely clear them.
Nix hacks for ignoring markers are restored and expanded so that we don't even
attempt to parse the marker.
Also drop preInstall cleanup for dependencies. The reason is while it's more
thorough than default cleanup in buildBazelPackage if such a problem happens we
should fix buildBazelPackage instead. Perhaps even move this (awesome!) snippet
there but it's very slow-running so we'd rather attempt to fix it in other
ways.
Anyway after an update at least .deps build, checked with `nix-build -A --check`.
Timestamp verification skip is no longer needed (not sure why). Generally we
better off always using the environment hack for all packages because that
ensures all NIX_* flags are correctly applied.
One possible improvement in future is to filter only NIX_* variables to
passthru in Bazel.
Without this Bazel always picks Python 3 which breaks Python 2 packages.
Strangely enough just dropping this patch works, with all `bazel.tests`
passing.
Without this Bazel always picks Python 3 which breaks Python 2 packages.
Strangely enough just dropping this patch works, with all `bazel.tests`
passing.
--distdir is now used in the installCheckPhase for prefetched
repositories. That's simpler, more robust and easier to extend in
the future.
Note that `name` argument of fetchurl was removed because it changed
the basename of the generated file and bazel uses this basename for
its cache behavior.
On Darwin, the last argument to GCC is coming up as an empty string.
This is breaking the build of proto_library targets. However, I was not
able to reproduce with the example cpp project[0].
This commit patches the cc_wrapper of Bazel that gets installed on
Darwin to remove the last argument if it's an empty string. This is
not a probem on Linux.
[0]: https://github.com/bazelbuild/examples/tree/master/cpp-tutorial/stage3
- Fixs for newly introduced bin/bash hardcoded reference
- Bazel now references `remote_java_tools_xxx` which contains prebuilt
binaries. We prefetch them, fix them, and force bazel to use the
fixed repository.
It also closes#63096
Whenever we create scripts that are installed to $out, we must use runtimeShell
in order to get the shell that can be executed on the machine we create the
package for. This is relevant for cross-compiling. The only use case for
stdenv.shell are scripts that are executed as part of the build system.
Usages in checkPhase are borderline however to decrease the likelyhood
of people copying the wrong examples, I decided to use runtimeShell as well.
By changing the default toolchain to JDK8, we broke the default Java
toolchain, which assumes JDK9.
Instead, set `host_java_toolchain` manually for our build of bazel,
and set `java_toolchain` to run the java tests with the build JDK as
well.
Fixes https://github.com/NixOS/nixpkgs/issues/54289
Since the 0.21 upgrade, the host `$PATH` is not forwarded anymore by
default to the sandboxes in charge to realize Bazel actions. This
default change broke the `py_binary` rule among other things.
Every python binary is wrapped in a stub in charge to setup the
execution environment. Currently, this stub's shebang points to a
`/usr/bin/env python` which cannot be resolved with the current
`$PATH`.
This results in breaking any build pipeline requiring the use of
python at some point. On top of the incorrect shebang, the stub
template is unable to find the actual python binary using
`SearchPath`.
This PR fixes those two things by re-writing the stub template shebang
to the actual python binary and by substituting the faulty default
python binary lookup to the right one.
0.21 removed the bundled openjdk-distribution. Instead, tries to fetch
the “right” distribution on-the-fly when building.
So we need to provide our own openjdk.
According to
https://github.com/bazelbuild/bazel/issues/6865#issuecomment-447261288
we should set `--host_javabase="@local_jdk//:jdk` if we want to do
that. This uses the jdk that is currently in the environment, which is
openjdk 8 in our case. 0.21 defaulted to a toolchain for JDK9, which
we don’t package in nixpkgs, so we use the JDK8 toolchain.
This commit also replaces the line-number-based sed invocations with
something more stable.
Bazel runs actions in a sandbox by default on Darwin and Linux.
However, the sandboxing was always and *silently* disabled previously,
because a Bazel feature test was always failing. The feature test
involved running `/bin/true` inside a sandbox. But on NixOS,
`/bin/true` does not exist...
This change is going to be required when upgrading to Bazel 0.20.0,
because in the checkPhase we're not wrapping the Bazel binary yet to
set some necessary default arguments.
Bazel supports per-workspace bootstrap scripts at $WORKSPACE_ROOT/
tools/bazel. This adds support for this behavior, which is needed
by many Bazel projects (OSS and private).
This finally fixes the build to avoid having to completely rebuild
bazel from source a second time just to generate the bash completion
script!
It also makes completion actually _work_ for bash users by
correcting the name of the installed script.
Bazel either reuses the `PATH` from the client, or sets a hardcoded
one. The former mode in problematic for build hermeticity. But the
latter is crippled on NixOS, because the hardcoded value is
`/bin:/usr/bin`. So we set the hardcoded value to match what
`customBash` provides. This has the effect of aligning the
environments for `ctx.actions.run` and `ctx.actions.run_shell`, which
were previously distinct (bug).
Bazel is a build tool, much like Make and many others. Like Make, it
should be agnostic to the compiler toolchains the user brings into
scope. Bazel has special rules that encode domain specific knowledge
for how to compile a C++ program, or indeed a Java program and a few
others. But that's not to say that at runtime Bazel should assume
a specific C++ compiler or Java compiler anymore than Make does.
The main impact of this change is that packages that build with Bazel
will have to list the compilers they want in their `buildInputs` or
similar, rather than relying on the `bazel` package pulling them in
transitively.
* Skip verifying checksums for already fetched packages.
Needed for two-staged building in Nix:
1. Build a fixed derivation with `bazel fetch` (filtered out of non-reproducable bits).
2. Build an actual derivation which uses fetched dependencies (skipping
checksums needed here because they depend on the build directory).
* Don't clean environment variables for children processes.
Needed for Nix compiler wrappers.
Removed patches that are purely for testing.
Removed dependencies that seemed to not be needed.
Expand all instances of #!/bin/bash, not just those at the start of scripts.
Bazel 981b7bc1 depends on protobuf-2.5 and won't work with 2.6 (and in
bbe84fe3d upgraded straight to protobuf 3.0.0-alpha3); this commit fixes
the dependency to depend on protobuf2_5 specifically.
The bazel compile.sh needs `which` on the PATH; so this commit adds that
as a dependency.
Setting JAVA_HOME to ${jdk} broke bazel when used with openjdk, with the
message:
Problem with java installation: couldn't find/access rt.jar in /nix/store/z9vc0vzyzhnpl5l5inmqdnvdnbxmmmg7-openjdk-8u60b24
This is because if you set JAVA_HOME, bazel will look for rt.jar in
$JAVA_HOME/lib and $JAVA_HOME/jre/lib, but the nixpkgs openjdk
distribution puts rt.jar in ${jdk}/lib/openjdk/jre/lib for some reason.
To fix this, this commit uses the ${jdk.home} passthru value to use the
appropriate JAVA_HOME for the given jdk.
As bazel now works with openjdk, and openjdk is free while oraclejdk is
not, this commit changes the default jdk for bazel to openjdk.
Since this package didn't have a listed maintainer, I'm claiming it.
