tensorflow assumes $USER to be set to something, otherwise it complains
like this:
```
FATAL: $USER is not set, and unable to look up name of current user: (error: 0): Success
Traceback (most recent call last):
File "./configure.py", line 1602, in <module>
main()
File "./configure.py", line 1399, in main
_TF_MAX_BAZEL_VERSION)
File "./configure.py", line 478, in check_bazel_version
['bazel', '--batch', '--bazelrc=/dev/null', 'version'])
File "./configure.py", line 156, in run_shell
output = subprocess.check_output(cmd)
File "/nix/store/drr8qcgiccfc5by09r5zc30flgwh1mbx-python3-3.7.5/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/nix/store/drr8qcgiccfc5by09r5zc30flgwh1mbx-python3-3.7.5/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
```
Spotted while changing the hash of its fixed-output derivation on
purpose.
We could also set this in the tensorflow-specific part, but very likely,
other programs will fail as well.
This cuts down the dependency tree on some rust builds where a crate not
just exposes a binary but also a library. `$out/lib` contained a bunch
of extra support files that among other information carry linker flags
(including the full path to link-time dependencies). Worst case this led
to some binary outputs depending on the full build closure of rust
crates.
Moving all the `$out/lib` files to `$lib/lib` solves this nicely.
`lib` might be a bit weird here as they are most of the time just rlib
files (rust libraries). Those are essential only required during
compilation but they can also be shared objects (like with traditional
C-style packages). Which is why I went with `lib` for the new output.
One of the caveats we are running into here is that we do not (always)
know ahead of time of a crate produces just a library or just a binary.
Cargo allows for some ambiguity regarding whether or not a crate
provides one, two, … binaries and libraries as it's outputs. Ideally we
would be able to rely on the `crateType` entirely but so far that isn't
the case. More work on that area might show how difficult that actually
is.
If an empty string is passed to `-idirafter`, it breaks gcc. This commit makes
the stdenv less fragile by expanding out the shell glob and ensuring no empty
arguments get passed.
Before, we'd always use `cc = null`, and check for that. The problem is
this breaks for cross compilation to platforms that don't support a C
compiler.
It's a very subtle issue. One might think there is no problem because we
have `stdenvNoCC`, and presumably one would only build derivations that
use that. The problem is that one still wants to use tools at build-time
that are themselves built with a C compiler, and those are gotten via
"splicing". The runtime version of those deps will explode, but the
build time / `buildPackages` versions of those deps will be fine, and
splicing attempts to work this by using `builtins.tryEval` to filter out
any broken "higher priority" packages (runtime is the default and
highest priority) so that both `foo` and `foo.nativeDrv` works.
However, `tryEval` only catches certain evaluation failures (e.g.
exceptions), and not arbitrary failures (such as `cc.attr` when `cc` is
null). This means `tryEval` fails to let us use our build time deps, and
everything comes apart.
The right solution is, as usually, to get rid of splicing. Or, baring
that, to make it so `foo` never works and one has to explicitly do
`foo.*`. But that is a much larger change, and certaily one unsuitable
to be backported to stable.
Given that, we instead make an exception-throwing `cc` attribute, and
create a `hasCC` attribute for those derivations which wish to
condtionally use a C compiler: instead of doing `stdenv.cc or null ==
null` or something similar, one does `stdenv.hasCC`. This allows quering
without "tripping" the exception, while also allowing `tryEval` to work.
No platform without a C compiler is yet wired up by default. That will
be done in a following commit.
Go beyond the obvious setup hooks now, with a bit of sed, with a skipped case:
- cc-wrapper's `dontlink`, because it already is handled.
Also, in nix files escaping was manually added.
EMP
Using wrapProgram adds a call to `bash` around every call
of `execline`, which clearly misses the basic idea behind
`execline` in the first place …
This reverts commit b64d25c447.
One issue with cargoSha256 is that it's hard to detect when it needs to
be updated or not. It's possible to upgrade a package and forget to
update cargoSha256 and run with old versions of the program or
libraries.
This commit introduces `verifyCargoDeps` which, when enabled, will check
that the Cargo.lock is not out of date in the cargoDeps by comparing it
with the package source.
Quoting from the splitString docstring:
NOTE: this function is not performant and should never be used.
This replaces trivial uses of splitString for splitting version
strings with the (potentially builtin) splitVersion.
There is a bug in this feature: It allows extra arguments to leak in
from the environment. For example:
$ export extraFlagsArray=date
$ man ls
Note that you get the man page for date rather than for ls. This happens
because 'man' happens to use a wrapper (to add groff to its PATH).
An attempt to fix this was made in 5ae18574fc in PR #19328 for
issue #2537, but 1. That change didn't actually fix the problem because
it addressed makeWrapper's environment during the build process, not the
constructed wrapper script's environment after installation, and 2. That
change was apparently accidentally lost when merged with 7ff6eec5fd.
Rather than trying to fix the bug again, we remove the extraFlagsArray
feature, since it has never been used in the public repo in the ten
years it has been available.
wrapAclocal continues to use its own, separate flavor of extraFlagsArray
in a more limited context. The analogous bug there was fixed in
4d7d10da6b in 2011.
This removes the unnecessary compiler build dependency. We also set
preferLocalBuild = true;
allowSubstitutes = false;
to not farm out the build on a remote builder or bother with trying to
find a binary substitution.
The architecture of an image should default to the architecture for
which that image is being composed or pulled. buildPackages.go.GOARCH is
an easy way to compute that architecture with the correct terminology.
These should not cause issues in practice but it is good idea to handle them.
* prefix and targetOffset are mandatory, as they are always set by the generic builder.
* wrapPrefixVariables and dontWrapGApps are now defaulting to empty value, as they are not mandatory.
Before this change, buildRustCrate always called rustc with
--extern libName=[...]libName[...]
However, Cargo permits using a different name under which a dependency
is known to a crate. For example, rand 0.7.0 uses:
[dependencies]
getrandom_package = { version = "0.1.1", package = "getrandom", optional = true }
Which introduces the getrandom dependency such that it is known as
getrandom_package to the rand crate. In this case, the correct extern
flag is of the form
--extern getrandom_package=[...]getrandom[...]
which is currently not supported. In order to support such cases, this
change introduces a crateRenames argument to buildRustCrate. This
argument is an attribute set of dependencies that should be renamed. In
this case, crateRenames would be:
{
"getrandom" = "getrandom_package";
}
The extern options are then built such that if the libName occurs as
an attribute in this set, it value will be used as the local
name. Otherwise libName will be used as before.
luarocks defines by default the following mirrors:
83093e7da7/src/luarocks/core/cfg.lua (L205)
Let's add them to nixpkgs. I have modified luarocks-nix to generate the
proper nixpkgs urls.
I bump luarocks-nix in the following commits.
This is a new package that provides a shell hook to make it easy to
declare manpages and shell completions in a manner that doesn't require
remembering where to actually install them. Basic usage looks like
{ stdenv, installShellFiles, ... }:
stdenv.mkDerivation {
# ...
nativeBuildInputs = [ installShellFiles ];
postInstall = ''
installManPage doc/foobar.1
installShellCompletion --bash share/completions/foobar.bash
installShellCompletion --fish share/completions/foobar.fish
installShellCompletion --zsh share/completions/_foobar
'';
# ...
}
See source comments for more details on the functions.
applyPatches applies a list of patches to a source directory.
For example to patch nixpkgs you can use:
applyPatches {
src = pkgs.path;
patches = [
(pkgs.fetchpatch {
url = "1f770d2055.patch";
sha256 = "1nlzx171y3r3jbk0qhvnl711kmdk57jlq4na8f8bs8wz2pbffymr";
})
];
}
There ver very many conflicts, basically all due to
name -> pname+version. Fortunately, almost everything was auto-resolved
by kdiff3, and for now I just fixed up a couple evaluation problems,
as verified by the tarball job. There might be some fallback to these
conflicts, but I believe it should be minimal.
Hydra nixpkgs: ?compare=1538299
They always can be regenerated during the actual build, and they are sometimes
random, e.g in Tensorflow;
platforms -> NIX_BUILD_TOP/tmp/install/35282f5123611afa742331368e9ae529/_embedded_binaries/platforms
This setup hook modifies a Perl script so that any "-I" flags in its shebang
line are rewritten into a "use lib ..." statement on the next line. This gets
around a limitation in Darwin, which will not properly handle a script whose
shebang line exceeds 511 characters.
Turns out markers are non-deterministic after all and even our patching still
doesn't solve this problem completely. For example (tensorflow deps, this is a
complete diff so actual dependencies don't differ):
30509c30509
< bc527ff00916b15caee38793bca8f294c748df4a256de55c5199281be0489e73 result/@bazel_skylib.marker
---
> 4e0303e815c78df1e43d4b88dfe65e73046e0c6157fb10aa9a4e8b910113cd9c result/@bazel_skylib.marker
31045c31045
< fa13d04b2316214c3b4008b52546c2d5b633e006f6f019d597bb3f9745bacf7b result/@bazel_toolchains.marker
---
> b36174bf5535e5157801b6de30c35ee03a03fe57766306393c3d65dd65cbebf4 result/@bazel_toolchains.marker
31144c31144
< b0ce4a3ac29ac22528336dd3a54b5b7af9ecc43bef2a2630713c1981a5cbbb51 result/@build_bazel_rules_swift.marker
---
> 7492528068ec4f8e7ace2ecf8f933ec4e1b2235bd7426ce6f70177919f1cd05e result/@build_bazel_rules_swift.marker
36245c36245
< be2993536a8233d63251b664caf35b1e7cd57d194ab2a39a293876c232d6bbd0 result/@io_bazel_rules_closure.marker
---
> b6655cc3f2c78525e5a724d8a4e93b1e7f09f1e09fc817d231109e7f39103e88 result/@io_bazel_rules_closure.marker
36329c36329
< 087bc674c9509dfe157400d111db4a13eeb45fc76aeccd490cee9aad6771ecad result/@io_bazel_rules_docker.marker
---
> f920ec07315ec71e800b05cd22b2a341c0a80807c6e335ee81739b13c532b422 result/@io_bazel_rules_docker.marker
79544d79543
< 85893a05a817036c61f6cd9f8247757baa1654f473c494ce4fc5253c2bbd2790 result/@platforms.marker
And here's an example of differences:
$ cat result-a/@bazel_skylib.marker
7dc7472d37424ba5ec6a5532765bc911
$MANAGED
cat result-b/@bazel_skylib.marker
a8f3f577798201157128e8e9934c4705
$MANAGED
Instead of trying to patch these markers further we now completely clear them.
Nix hacks for ignoring markers are restored and expanded so that we don't even
attempt to parse the marker.
* buildBazelPackage: autodetect nix toolchain instead of Xcode on Darwin
* do not export the variables outside of Darwin
* remove unecessary parens
* move comment within the darwin check
Timestamp verification skip is no longer needed (not sure why). Generally we
better off always using the environment hack for all packages because that
ensures all NIX_* flags are correctly applied.
One possible improvement in future is to filter only NIX_* variables to
passthru in Bazel.
Before one would get the following error
nix-repl> pkgs.writeTextDir "share/my-file" "foo"
error: invalid character '/' in name 'share/my-file'
Fixes#50347
Previously the installPhase of the fixed ouput derivation would fail for
a package that has no markers, since `sed` would complain about having
no input files. If we use `find` instead of bash globs, that problem
goes away.
This avoids dumping -Wall warnings when they appear in framework
headers. As a result, we are closer to how regular headers are
included (via -isystem).
Also remove ccIncludeFlag lookup, this was unused & not very useful.
mergeInputs is now simply defined in terms of `concatLists` and
`catAttrs` instead of a more complicated `foldr`.
Note that the order of PATH has also changed. For example running the
following with nix-shell:
let
pkgs = import <nixpkgs> {};
shell1 = pkgs.mkShell {
buildInputs = [ pkgs.htop ];
};
shell2 = pkgs.mkShell {
buildInputs = [ pkgs.hello ];
};
shell3 = pkgs.mkShell {
inputsFrom = [ shell1 shell2 ];
buildInputs = [ pkgs.tree ];
};
in shell3
Results in the following PATH:
$ echo $PATH
...
/nix/store/yifq4bikf7m07160bpia7z48ciqddbfi-tree-1.8.0/bin:
/nix/store/vhxqk81234ivqw1a7j200a1c69k8mywi-htop-2.2.0/bin:
/nix/store/n9vm3m58y1n3rg3mlll17wanc9hln58k-hello-2.10/bin
...
Previously the order was:
/nix/store/n9vm3m58y1n3rg3mlll17wanc9hln58k-hello-2.10/bin
/nix/store/vhxqk81234ivqw1a7j200a1c69k8mywi-htop-2.2.0/bin:
/nix/store/yifq4bikf7m07160bpia7z48ciqddbfi-tree-1.8.0/bin:
I think the new order makes more sense because it allows to override
the PATH in the outermost mkShell.
Running the following expression with nix-shell:
let
pkgs = import <nixpkgs> {};
shell1 = pkgs.mkShell {
shellHook = ''
echo shell1
'';
};
shell2 = pkgs.mkShell {
shellHook = ''
echo shell2
'';
};
shell3 = pkgs.mkShell {
inputsFrom = [ shell1 shell2 ];
shellHook = ''
echo shell3
'';
};
in shell3
Will now results in:
shell2
shell1
shell3
Note that packages in the front of inputsFrom have precedence over
packages in the back. The outermost mkShell has precedence over all.
The previous behaviour would work fine as long as `symlink` is a link to
a file. If is a link to a directory though, the new `ln` wouldn't
overwrite it but would create a new link *in that directory* (with the
name of the link source).
Instead, we can precompute the target location, then first remove the
symlink and write the new one in its place.
After bumping sublime3 in #61636 we realized that saving files as root
doesn’t work anymore and somehow the paths weren’t patched by
`libredirect`.
After some debugging it came out that Sublime switched from `posix_spawn(3)`
to `posix_spawnp(3)` to start new processes internally. Since `libredirect`
only handled the former, `/usr/bin/pkexec` stopped being redirected.
Wrapping `posix_spawnp` fixes the problem.
PR #58431 added /nix/store to each layer.tar. However, the timestamp was
not explicitly set while adding /nix and /nix/store to the archive. This
resulted in different SHA256 hashes of layer.tar between image builds.
This change sets time and owner when tar'ing /nix/store.
To avoid symlink loops to /host in nested chrootenvs we need to remove
one level of indirection. This is also what's generally expected of
/host contents.
* Remove unused argument from pivot_root;
* Factor out tmpdir creation into a separate function;
* Remove unused fstype from bind mount;
* Use unlink instead of a treewalk to remove empty temporary directory.
The problem with stacking chrootenv before was that CLONE_NEWUSER cannot
be used when a child uses chroot. So instead of that we use pivot_root
which replaces root in the whole namespace. This requires our new root
to be an actual fs so we mount tmpfs.
not all valid file names are valid derivation names. This can cause
troubles when, for example, trying to place systemd template unit
files, which contain an '@' in their name, in an initrd.
Fixes#53987
glibc 2.27 (and possibly other versions) can't handle an `nopenfd` value larger than 2^19 in `ntfw`, which is problematic if you've set the maximum number of fds per process to a value higher than that.
Commit "patchShebangs: Allow for multiple arguments" 4a1e51f957
removed the check. We don't want to break existing usages so this
introduces it again with a successful exit code.
It's tempting to think patchShebangs supports multiple arguments.
Without this patch it just silently ignores all but the first. Now it
patches the shebangs in all of its arguments.
Fixes: #57695
This hook allows to add NixOS driver libraries path to given ELF
objects' RUNPATH. We use it instead of settings RUNPATH manually
everywhere. It must be invoked in postFixup so that RUNPATH stripping
does not remove the path.
It puts the path first instead of last so that system-wide drivers
are always preferred.
The layer order was not correct when a parent image was used: parent
image layers were above the new created layer.
This commits simplifies the code related to layer ordering. In
particular, layers in `layer-list` are ordered from bottom-most to
top-most. This is also the order of layers in the `rootfs.diff_ids`
attribute of the image configuration.
While it might be useful in some cases, there are too many caveats to be worth it.
When libredirect intercepts dlopen call and calls the original function, the dynamic
loader will use libredirect.so's DT_RUNPATH entry instead of the one from the ELF file
the dlopen call originated from. That means that when program tries to dlopen a library
that it expects to find on its RPATH, the call will fail.
This broke Sublime Text for just that reason.
Especially as a new user it is a much better experience to receive a
proper help response to `-h`. Currently passing `-h` will cause some
runtime error with the `git remote` error help being shown. Not very
helpful.
It doesn't hurt to be a bit more user friendly in this case.
This introduces extra escaping for $NIX_STORE that gets interpolated
into a PCRE. The escaping is performed using a standard Perl function
"quotemeta" (see "perldoc -f quotemeta" for reference). The same value
is also used in sed regex which uses POSIX basic regular expressions
instead of PCRE, so it needs fewer characters to be escaped. It should
not cause much problem to not change sed invocation, but I replace it
with equivalent Perl expression (actually the behavior is changed to not
output a newline character after the matched output).
